FX filter continue to be applied after removing texture

If I select texture to bitmap and add a filter (e.g black and white). Then I remove the texture (i.e. Back to a solid colour) then the filter (in this case black and white) continues to be applied. Resulting in a shade of grey instead of the selected colour. The filter can then only be removed by selecting bitmap in texture again, removing the filter, the removing the texture.