I've gone as small as 4 1/4, but have had much more success with larger grip sizes. I really think it's what you're used to, because so many people start out now with smaller grips. 4 5/8 with an overgrip feels most natural to me, though I can go down to bare 4 1/2. I think it's a matter of taste, but I feel like I'm in more control of the racquet at that size - probably because I'm trying to over-muscle the racquet. Still, when I use a smaller grip, I find it even harder to relax my grip.
I have come to the conclusion that I simply don't know what I don't know.