It doesn't seem that hard to understand to me. Someone who carries around singles sticks all the time and always plays with them is used to the net at that (proper) height. Someone else who never plays using singles sticks is not used to the net at that height. This certainly could, in theory, be an advantage for the guy who carries around singles sticks, however large or small that advantage may be. The other player could simply not wish for the match to be played under conditions which singles sticks guy is more used to than he is.
I wouldn't personally decline if someone wanted to use them, but at the same time the rational of why someone might decline is not terribly difficult to understand either, imo.
It's reasonable to conclude that using singles sticks would be a disadvantage to the opponent who is not used to them. So it's reasonable that given the option, the opponent would decline to use them.
However, should they be allowed to decline them? That is the question!
Here's an analogy: Let's say that I play only with brand X balls - they are really crappy and are solid black! But they are cheap! Needless to say, they are not approved for play under the Rules of Tennis.
You show up at our flex match with normal balls. Can I refuse to use the normal balls, seeing as I would be at a disadvantage because they are different to what I'm used to? Can I make you use my Black-X balls? I think most would say I have no choice but to use the regular balls, since those are the ones accepted for common usage, and approved in the rules.
So I think it comes down to what is the general norm, what is considered acceptable, and what the rules mandate. Usually common practice and the rules are aligned, so there is no conflict.
But with singles sticks, the rules and general norm are in contradiction. The rules explicitly require the use of singles sticks:
Friend at Court - ITF Rules of Tennis - 1.The Court:
"For singles matches, if a singles net is used, the centres of the net posts shall be 3 feet (0.914 m) outside the singles court on each side. If a doubles net is used, then the net
shall be supported, at a height of 31⁄2 feet (1.07 m), by two singles sticks, the centres of which shall be 3 feet (0.914 m) outside the singles court on each side."
The bolding is mine - it seems clear that the use of singles sticks is mandated, unless a dedicated singles net is being used. There is no provision anywhere in Friend at Court or otherwise in the Rules to excuse the use of singles sticks.
However, they are not commonly used. So what takes precedence - the rules, or common usage? That is the question...