As you touched on, one reason to buy racquets with small handles is because it's easier to build a grip up in size than to shave it down if it's too large.
Agreed. That is probably the main reason why people tend to buy racquets with smaller grips.
I wonder though, why some of the pros use smaller grips. Federer's grip size is probably one or two sizes smaller than TW's rough recommendation seems to indicate. Nadal's grip is even smaller. Is it because smaller grips are easier when it comes to hitting "whippy" shots by bringing the wrist into play? Smaller grips may cause tennis elbow, but the pros are so powerful and train so diligently that most of them needn't worry about these things.
If I use TW's measurement, my grip size tends to be between 4 3/8 and 4 1/2, but I'm always more comfortable with 4 1/2. When I last played with 4 3/8 grip, I felt that the "hit-through" component of my shot suffered, while the "bursh-up" component improved. It was a PK 5g, which is not a very powerful racquet, so I had a hard time clearing the service line on my groundstrokes with a 4 3/8 grip.