I've had frostbite (twice) of my fingers - mountain climbing. Anything much below 60 degrees F and my fingers/hands (which always sweat a little) start to get too stiff and cold to handle the racquet - not a lot but enough to make playing difficult. Even a light breeze makes it worse. Once things get in the low 50s I start sticking my hands in my pockets and I wear gloves starting below 50 - before most people feel they need gloves. Below freezing I wear mittens with liner gloves. Even playing indoors in sometimes chilly tennis clubs, I carry chemical heat packs which I constantly use to keep my hands warm.
It's kind of a drag because it somewhat truncates the outdoor season for playing tennis.