Do you get annoyed with people who say they have allergies and make out that they’re going to die if they come anywhere near a tiny bit of gluten but then it turns out it’s only a mild intolerance and they stuff their faces with Yorkshire puddings?
It's usually not even that. "I'm allergic to" is usually synonymous with "I don't like". It's annoying because it devalues the meaning for people that are truly, life threateningly allergic.