if the guys find it offensive, they're not paying enough attention to their own game. there is nothing offensive about this.
i sometimes wear a sweater or jacket and shorts if i'm playing in the cold (rarely happens). Usually take it off after the first set, or a few games once i warm up.
Disclaimer: I'm NOT a coach...
Real tennis: https://www.youtube.com/watch?v=yDqnkLJ9BtM