Its tough, but as the net man on the returning team, just watch the service line. Don't try to watch it all the way from the server to the court, you'll miss close calls. If you watch the line, you can almost always get a good call. Once the ball is in, refocus on your preferred target.
"Outside the door, he took four more. What a drag it is getting old. He goes running, for the shelter, of Mallorca's Little Helper."