One thing I noticed when using a grip that was too small for me was that the racquet was more easily knocked around in my hand.
You really have to play test different sizes in my opinion. If you go too small, you can build it up with additional over wraps or whatever. You have to be careful not to build it up so much as to lose the feel of the bevels. If you go too big, well that's a different story.
NTRP 3.5 with < 130mph TTW serve | Head Microgel Radical MP (3g split at 3&9, 5g in handle) | Babolat Xcel French Open 16ga @ 58 | Year 2 of playing