Fuji, IME there is a lot of difference between the serve of a 4.0 guy and a 4.0 gal.
In ladies, I hardly ever face a server who has a serve I will struggle to return. Most of the first serves are flat and come nice and straight. Most of the second serves are a modified push -- just a slower version of the first -- or are puffballs.
The 4.0 guys will have more pace on their flat serve. But the biggest difference is that the 4.0 guys can use spin to confound or to yank you off the court. So if I am receiving on the ad court and the guy hits a slice up the middle, there's just no way I can get enough on the return to go at the woman or down her alley. Gotta send it right back to the guy server.
-- Random Error Generator, Version 4.0
-- Master Moonballer