It is ok to notice the toss and its location but you should probably not dwell on it too much. If I recall correctly, gaze control studies show that elite players focus primarily on the upward swing (path) of the server's racquet. It was found that eyes of lesser skilled players tended to dart around much more than elite players. Elite players kept their eyes quieter -- they did not necessarily follow the toss up and own as one might expect. Less is more. Try focusing on the upward swing more and see if that doesn't help.
Every tool is a weapon -- if you hold it right. (~Ani DiFranco)