This kind of goes along the lines of if someone shows up dresses like Federer and has his equipment as well, but obviously can't play like him.
Bottom line is this. Wear what you want, play with what you want, play however you want. If anyone judges you, tell them to f o.
Chances are good they've got something that they use because a pro uses it somewhere down the line as well, so who's the pot calling the kettle black now?
As for me, I keep my long pants on as well as a long sleeve shirt when the temps are under 50 out. Anything like that and even if I'm hot, I leave them on because if I take them off, my muscles would eventually get cold due to wind/whatever. Either way, if you're wearing perf materials you shouldn't be dripping wet anyway, even if you're warm. It's better to be warm and not pull something than to be cold and pull a muscle. Yes, it can happen... and very easily in the winter times.