navigator
Hall of Fame
I've been going to the UTR website every couple of months for the last year to see how ratings were shaking out among various folks that I know that play in different "classifications" (NTRP, senior, college, junior, etc). I've always thought NTRP was kind of silly and I was hopeful regarding UTR, but alas... it's got some major problems. Here are a few observations:
(1) The ratings are wildly inaccurate across classifications. For example, if you compare men that only play seniors events with boys that only play junior events, the ratings make almost no sense whatsoever. (And the players within these classifications almost never cross paths in actual tournaments.) Likewise, for example, if you compare the men that only play NTRP events with men that only play senior events, again the comparative ratings are way off.
A few examples. The guy who just won the over-50s World Championships has a 9.8 UTR. (He has an "official" or "complete" rating due to the high number of matches he's played.) Two U14s that I play and beat easily have (official) UTRs of 10+. The over-50s World Champion would beat me 60 60 (or thereabouts) in a typical match. These two U14s would struggle to get more than a few points each set. And yet they have higher (official) UTRs. So, this is way way way off... and not at all anecdotal (I could provide a lot more similar examples). Likewise, I know of a few guys that play almost solely NTRP events that have much higher UTRs than guys that would beat them, but who only play seniors events. Which leads to...
(2) The relative ratings appear to be roughly accurate within classifications. So, for example, when I look at a number of guys that I know who only play seniors events, their relative ratings aren't too far off. The better players tend to have higher UTRs roughly commensurate with their higher skill, and vice versa. Likewise in the case of guys that only play NTRP events - their UTRs relative to others that only play NTRP events look about right. The better players tend to have the commensurately higher UTRs. This appears to be the case with the juniors, college players, etc.
But once you try to compare, for example, junior boys with junior girls, or NTRP adults with senior players, or college players with junior players, etc... the whole thing goes to pot. The UTRs just don't make much sense. But... this should not be surprising because... these various groups rarely cross paths in tournaments. Perhaps if more folks start playing UTR tournaments then the ratings across these groups will improve, but... I don't know, I just don't see it happening.
(3) Too many "official" ratings are based on matches with players who don't have "official" ratings. I've mentioned this before - it's a "junk in, junk out" problem. If your "official" rating is based on matches largely played against opponents with "incomplete" ratings, then... that "official" rating is junk. For juniors who play a lot of tournaments or college players who play a lot of matches, etc, the statistical stew is pretty rich and the relative UTRs should be pretty good. But for the vast majority of players in the database... they're not so good.
Anyhow, I've always thought NTRP was pretty flawed. But it appears that UTR - at least when comparing across groups - is actually worse. I'll be interested to see if they can fix it.
[By the way, this guy did some good work on UTR with high school players and found it to be quite accurate. (http://alexslezak.com/2014/09/19/how-accurate-is-the-universal-tennis-rating-system/) This doesn't surprise me at all because... he limited his analysis to a single classification - male high school tennis players. He didn't try to compare across classifications - to include girls, for example.]
(1) The ratings are wildly inaccurate across classifications. For example, if you compare men that only play seniors events with boys that only play junior events, the ratings make almost no sense whatsoever. (And the players within these classifications almost never cross paths in actual tournaments.) Likewise, for example, if you compare the men that only play NTRP events with men that only play senior events, again the comparative ratings are way off.
A few examples. The guy who just won the over-50s World Championships has a 9.8 UTR. (He has an "official" or "complete" rating due to the high number of matches he's played.) Two U14s that I play and beat easily have (official) UTRs of 10+. The over-50s World Champion would beat me 60 60 (or thereabouts) in a typical match. These two U14s would struggle to get more than a few points each set. And yet they have higher (official) UTRs. So, this is way way way off... and not at all anecdotal (I could provide a lot more similar examples). Likewise, I know of a few guys that play almost solely NTRP events that have much higher UTRs than guys that would beat them, but who only play seniors events. Which leads to...
(2) The relative ratings appear to be roughly accurate within classifications. So, for example, when I look at a number of guys that I know who only play seniors events, their relative ratings aren't too far off. The better players tend to have higher UTRs roughly commensurate with their higher skill, and vice versa. Likewise in the case of guys that only play NTRP events - their UTRs relative to others that only play NTRP events look about right. The better players tend to have the commensurately higher UTRs. This appears to be the case with the juniors, college players, etc.
But once you try to compare, for example, junior boys with junior girls, or NTRP adults with senior players, or college players with junior players, etc... the whole thing goes to pot. The UTRs just don't make much sense. But... this should not be surprising because... these various groups rarely cross paths in tournaments. Perhaps if more folks start playing UTR tournaments then the ratings across these groups will improve, but... I don't know, I just don't see it happening.
(3) Too many "official" ratings are based on matches with players who don't have "official" ratings. I've mentioned this before - it's a "junk in, junk out" problem. If your "official" rating is based on matches largely played against opponents with "incomplete" ratings, then... that "official" rating is junk. For juniors who play a lot of tournaments or college players who play a lot of matches, etc, the statistical stew is pretty rich and the relative UTRs should be pretty good. But for the vast majority of players in the database... they're not so good.
Anyhow, I've always thought NTRP was pretty flawed. But it appears that UTR - at least when comparing across groups - is actually worse. I'll be interested to see if they can fix it.
[By the way, this guy did some good work on UTR with high school players and found it to be quite accurate. (http://alexslezak.com/2014/09/19/how-accurate-is-the-universal-tennis-rating-system/) This doesn't surprise me at all because... he limited his analysis to a single classification - male high school tennis players. He didn't try to compare across classifications - to include girls, for example.]