I hadn't really given this much thought, but I actually was involved in this situation over the weekend.
I went easy on the serves for most of the match, but I hit them harder and flatter towards the end of the 2nd set (to make sure we won a close one). I don't have a monster serve, but it's fast enough. I didn't even think of easing up on the woman (who was good enough).
Maybe they thought I was a ******?