Oddly enough I hear "I don't do math" or variations quite a bit. My latest is to respond, "but people will do math that affects you" - emphasize that you might not be using it, but others will in a way that can harm you/rip you off etc. Seems to work somewhat.
I hear the same, though not about math (well...about math also). When someone tells me they don't like politics or don't vote or don't care to be informed, I ask them to consider who benefits from their complacency...it probably isn't someone with their best interests in mind.