Your question is a false dichotomy ("but if I then told you that 2 + 2 = 5 what would you think? Would you simply say I was going against popular opinion or against reality?").
I would not say either one of those statements. I would simply ask you to explain.
(IFF) "because numbers are not associated with units like apples" (THEN) how could "2 + 2 = 5" be "against reality"?
Wouldn't it be more precise to say, "it's apparently against the purely abstract rules of mathemajicks?"
YOur "question" cannot be answered unless you make your definitions explicit.
You're the one making the claim "2 + 2 = 5", not me.
YOu need to explain yourself (support your own claim).