When The Ratings Go Wrong!?!? (Share Your Best Example)

Jack the Hack

Hall of Fame
Every year when the new ratings come out, there are always some crazy cases where individuals get unjustly bumped up or down.

Now that the new 2007 ratings have been published, please share the most outrageous examples you've found where the NTRP computer got it wrong. I'll start:

Brian Joelson (who just recently turned 46) played at Wimbledon in 1993, losing a close 5 setter to Amos Mansdorf in the 1st round. He played the challenger circuit for many years, and was ranked as high as #221 in singles, and #142 in doubles. He made 4 challenger level finals in doubles, and also played doubles at Wimbledon and the Australian Open. Here is his ATP profile:

http://www.atptennis.com/3/en/players/playerprofiles/?playernumber=J036

He was also a two-time Oregon state high school champion in singles, and an All-American at Texas A&M University. Between 1985-1999, Joelson was the #1 ranked singles player in the Open category in the Pacific Northwest seven times, and a five time winner of the section's "Player of the Year" award. In addition, Brian and his son Brett won the National Father and Son Hard Court Championships four years in a row from 2003-2006, and are the #1 seeds in the 2007 event (which starts on Friday).

Here's another profile of him, along with a picture:

http://www.irvingtonclub.com/tho/players_joelson.html

Joelson is also in tremendous shape, and looks like he's still in his late 20s age-wise. Playing in Open tournaments this year, he went 7-3 in singles and 16-5 in doubles. In his singles wins, five of them were over strong 5.0 players, and his losses were all to 5.5 players with ATP rankings. In doubles, he won a 45 and over tournament with his brother, won an Open tournament with his son, and made the finals of one of the PNW pro circuit tournaments with Mark Quinney (another former PNW #1, and brother to professional golfer Jeff Quinney). All of his losses in doubles were to teams of 5.0 minimum, and several had 5.5 or 6.0 rated players.

Here is his TennisLink record:

http://tennislink.usta.com/leagues/...0060D157E0215D85D43B7A49D83E2F1C0D&CYear=2007

So...

What do you think the mighty NTRP computer will generate as a rating for Mr. Joelson?

Let's try 4.5:

http://tennislink.usta.com/leagues/...son&FirstName=brian&SearchType=X&submit2=+Go+

Top that!!! :shock:
 
Going from the data I crunched in this thread (http://tt.tennis-warehouse.com/showthread.php?t=166618), I think I've gotten to the point where I think it's just plain hard to handicap really good players.

Of course, this begs the question of what constitutes a "really good" player.

If we go by the data, 4.5 represents the top 85th to 95th percentiles of tennis players participating in USTA leagues, give or take. 5.0 goes from the 95th to the 99.5th. 5.5 from 99.5 to (you're going to love this) 99.9.

I realize population percentiles don't correspond exactly to skill level, but is this guy in the top 0.1% of USTA league participants? No. So he's not a 6.0.

How about the top 0.5%? Maybe, but I could buy that he isn't, so he's not a 5.5, either.

How about the top 5%? I'd say yes. The USTA says no. Sucks for weak 4.5s, but them's the breaks.

Obviously, one of the problems here is the tails on the USTA bell curve aren't nearly fat enough, with a giant log jam of players between 3.5 and 4.0 (20th-85th). By making the air at 5.0+ so rarefied, anybody failing to live up to the algorithm at that point drops down to do battle with the mucks that are pretty darn good, but not good enough to do without handicapping.

Speaking personally, I just found out a guy on my team just got bumped, with a worse winning percentage, against opponents with a lower aggregate winning percentage than the dudes I faced.

As always - who can truly understand what the USTA does?
 
Going from the data I crunched in this thread (http://tt.tennis-warehouse.com/showthread.php?t=166618), I think I've gotten to the point where I think it's just plain hard to handicap really good players.

Of course, this begs the question of what constitutes a "really good" player.

If we go by the data, 4.5 represents the top 85th to 95th percentiles of tennis players participating in USTA leagues, give or take. 5.0 goes from the 95th to the 99.5th. 5.5 from 99.5 to (you're going to love this) 99.9.

I realize population percentiles don't correspond exactly to skill level, but is this guy in the top 0.1% of USTA league participants? No. So he's not a 6.0.

How about the top 0.5%? Maybe, but I could buy that he isn't, so he's not a 5.5, either.

How about the top 5%? I'd say yes. The USTA says no. Sucks for weak 4.5s, but them's the breaks.


Obviously, one of the problems here is the tails on the USTA bell curve aren't nearly fat enough, with a giant log jam of players between 3.5 and 4.0 (20th-85th). By making the air at 5.0+ so rarefied, anybody failing to live up to the algorithm at that point drops down to do battle with the mucks that are pretty darn good, but not good enough to do without handicapping.

Speaking personally, I just found out a guy on my team just got bumped, with a worse winning percentage, against opponents with a lower aggregate winning percentage than the dudes I faced.

As always - who can truly understand what the USTA does?

How many 4.5 or 5.0 players have played at Wimbledon in the past 15 years?

How many 4.5 or 5.0 players have been ranked #1 in the Open category in their section for numerous years, and have recently won a national title?

I would say the answer is less than 0.5%.

I could (maybe) feasibly see a player like Joelson dropping down to 4.5 if they had gotten horribly out of shape, stopped playing the game for many years, or had become handicapped in an accident. However, the fact is that Joelson never stopped playing and is in incredible shape.

I think his playing experience puts him into the top 1% of all USTA League participants, which means that he is in the upper ranges of the 5.0 category, nearly a 5.5. His tournament results this year reflect this. There is not a single 4.5 player in the US that could take a set off him, let alone win a match. (Even the sandbaggers wouldn't win.)
 
I don't think you can protest a computer rated player. Admittedly, I am not really sure.

No reason to protest unless Joelson actually tries to play on a 4.5 team (which I doubt he will). Also, I believe you are correct in that you are not supposed to be able to protest a computer rated player. However, in this case, I think it shows an obvious error in the USTA system.

I'm actually considering whether to write to the section and national office about this case... I wonder what their reaction would be?
 
The system is obviously flawed.

For the most part, I think it does a pretty good job; yet, it can be manipulated. If you do not manipulate the system, it is hard to get a team to nationals (much less win).
 
How many 4.5 or 5.0 players have played at Wimbledon in the past 15 years?

How many 4.5 or 5.0 players have been ranked #1 in the Open category in their section for numerous years, and have recently won a national title?

I would say the answer is less than 0.5%.

I could (maybe) feasibly see a player like Joelson dropping down to 4.5 if they had gotten horribly out of shape, stopped playing the game for many years, or had become handicapped in an accident. However, the fact is that Joelson never stopped playing and is in incredible shape.

I think his playing experience puts him into the top 1% of all USTA League participants, which means that he is in the upper ranges of the 5.0 category, nearly a 5.5. His tournament results this year reflect this. There is not a single 4.5 player in the US that could take a set off him, let alone win a match. (Even the sandbaggers wouldn't win.)

You're preaching to the choir, here. In my opinion, he's really got no business playing even 5.0 or 5.5, but I'm not the USTA.

I'll say this much - seeing the population numbers has REALLY opened my eyes to either how broken the NTRP is, or how ignorant the general population is, or both. Any large tennis scene has lots of people thinking they belong at 4.5, but given that those players now represent the top 15% of the playing population and even guys with open titles and grand slam experience, there's really not much more to do but laugh.
 
LOL .. there are many players out there that are severely under-rated, not to mention the players that beat them ;) The NTRP algorithm has a bad habit of pushing people to the "center," which would be 3.5 and 4.0.

So unless they benchmark former ATP and WTA professionals, maybe create a "pro" benchmark to only play in "Open" league, then you'll see a lot of 6.0's playing as 4.5.

A year or so ago, a former WTA professional who played 3.5 tournaments and then self-rated 4.0 was finally "fixed" to 6.0 by Nationals after a lot of protests. But it takes a lot to get someone back up there.
 
No reason to protest unless Joelson actually tries to play on a 4.5 team (which I doubt he will). Also, I believe you are correct in that you are not supposed to be able to protest a computer rated player. However, in this case, I think it shows an obvious error in the USTA system.

I'm actually considering whether to write to the section and national office about this case... I wonder what their reaction would be?

How could he even get a tournament 4.5, he beat a 5.0 player 1,0
Also in 2003 he won 2 matches at 5.5 nationals.
Plus he played one 10.0 MXD match, this has to be a joke.
 
The system is obviously flawed.

For the most part, I think it does a pretty good job; yet, it can be manipulated. If you do not manipulate the system, it is hard to get a team to nationals (much less win).

I went back over all of Joelson's results since 2003. I could not find one single instance where he lost to a player rated lower than 5.0. In fact, I could hardly find any cases where he even played somebody rated below 5.0. During this timeframe, he compiled a record of 188-50 (79% winning percentage) and won 23 titles, almost all of them in the Open category.

He made no attempt to "manipulate" his rating by throwing matches or dodging competition. I simply can find no reason why the computer would generate a 4.5 rating for this player.

Raiden.Kaminari said:
The NTRP algorithm has a bad habit of pushing people to the "center," which would be 3.5 and 4.0.

That is the only explanation for what is happening in this case.
 
How could he even get a tournament 4.5, he beat a 5.0 player 1,0
Also in 2003 he won 2 matches at 5.5 nationals.
Plus he played one 10.0 MXD match, this has to be a joke.

Unfortunately, if Joelson does decide to play for a 4.5 team and the USTA lets this stand, the joke is on us.

Frankly, I don't think he cares enough about USTA League to mess around with the 4.5 level. On the other hand, one of his good friends (who was also on that 5.5 Nationals team in '03) also got rated down to 4.5 this week. If they decide to play together on one of the local power teams, they would be legitimate favorites to win the National title pretty easily.
 
I recently played doubles with a player who said that a couple years ago he self-rated at 3.5 and played in a 7.0 mixed league. He played like 2 or 3 matches and I think he lost all of them, and was bumped up to 4.5 in mixed exclusive ratings. The USTA refused to drop him lower and he quit and never played league since. He's definitely not a 4.0 or above.
 
There are flaws with any system. My example (by no means as eggregious as yours).

Buddy of mine, bumped from 3.0 to 4.0 after winning 4 matches at 3.5. Doesn't play the rest of the year, is now a benchmark player. Now I've beat him 0-1-0 and I'm a strong 4.0 but get handled at 4.5 by anything but the lower 25% of 4.5's.

How you register, play 4 matches, go from 3.0 to 4.0, and become a benchmark is a little odd to me.
 
Wow, that's nuts!

jack, thanks for taking the time to look all those links up and post.

No problem. Me and one of my buddies are both TennisLink junkies. We are trying to put together a 4.5 team that will have a shot at going to Sectionals, or maybe even Nationals... so we were scouring the ratings for good players in the area that might have dropped down from 5.0 yesterday. When we saw Joelson listed as a 4.5, we just about fell over!!!

(Hmmm... maybe we can get him to play for us... :twisted:)

By the way, if this guy is a 4.5, maybe there really are 3.5s with 6 foot kick serves??? ;)
 
LOL .. there are many players out there that are severely under-rated, not to mention the players that beat them ;) The NTRP algorithm has a bad habit of pushing people to the "center," which would be 3.5 and 4.0.

So unless they benchmark former ATP and WTA professionals, maybe create a "pro" benchmark to only play in "Open" league, then you'll see a lot of 6.0's playing as 4.5.

A year or so ago, a former WTA professional who played 3.5 tournaments and then self-rated 4.0 was finally "fixed" to 6.0 by Nationals after a lot of protests. But it takes a lot to get someone back up there.

I just checked the NTRP of the former WTA professional. She's apparently back down to 4.5 as well.

Now I don't feel so bad, but I was half expecting to get bumped even lower.
 
I don't think you can protest a computer rated player. Admittedly, I am not really sure.

I know someone that played in a pro tour in Europe. Every year, during the early start ratings, he's listed as a 4.5. But a day later it is changed back to a 5.0. I've heard that someone has complained and the State Office manually moves him up everytime.

Don't know if that's true, but I have seen his rating a 4.5 on one day then 5.0 the next. This was prior to instand appeals, so the only way it would change is if someone manually changed it.
 
I know someone that played in a pro tour in Europe. Every year, during the early start ratings, he's listed as a 4.5. But a day later it is changed back to a 5.0. I've heard that someone has complained and the State Office manually moves him up everytime.

Don't know if that's true, but I have seen his rating a 4.5 on one day then 5.0 the next. This was prior to instand appeals, so the only way it would change is if someone manually changed it.

Well, old Brian whatzisname is up to 5.0 this morning, too ....
 
By the way, if this guy is a 4.5, maybe there really are 3.5s with 6 foot kick serves??? ;)

Funny enough, I play 3.5 right now, and I can hit a kicker that bounces to the shoulders of most average height men (probably a 5 foot or so kicker). ;-)

My usual dubs partner is worse, though - he went to the Pacific Life Tri-level last year and had a serve clocked at 122. :shock: He's going to get bumped to 4.0 this year, though, I'd bet.
 
My husband and I are both rated 3.5 and play 7.0 combo mixed doubles. We played a match last October and the guys serve bounced over my head everytime most of the time to my backhand. I was so mad after the match I didn't want to talk to them because I thought he was playing down so badly. I checked his record later and sure enough he was a 3.5 and I felt pretty badly about not wanting to talk to them after the match. I'm 5'4" so I'm not sure if it kicked up 6' but close to it.
 
Well, old Brian whatzisname is up to 5.0 this morning, too ....

Yep... you better believe there was a huge outcry within the Pacific Northwest Section when everybody saw that 4.5 rating come out!

The Joelson family is very well known within the section. Aside from all of the accomplishments that I listed in my first post, Joelson's son Brett was the #1 ranked Open player in the section last year and his daughter Ashley plays for UCLA. Brian's brother Brad also plays a lot of tournaments in the Northwest and also had an Open ranking.

I know that there were hundreds of people that went "WHAAAAAAAATTTT?" when Brian got listed as a 4.5, and the USTA was nailed with e-mails. Thankfully for them, they made the appropriate change very quickly. The question that the USTA needs to answer is how the computer could have possibly generated a 4.5 rating for him in the first place?

(By the way, Joelson's good friend that was also on the '03 5.5 Nationals team is still listed as a 4.5, so I'm going to start gathering info on him next. I know his friend was ranked #1 in the PNW Open doubles rankings twice, but he doesn't have any former ATP ranking or Grand Slam experience, so his case is a lot less outrageous.)
 
Funny enough, I play 3.5 right now, and I can hit a kicker that bounces to the shoulders of most average height men (probably a 5 foot or so kicker). ;-)

My usual dubs partner is worse, though - he went to the Pacific Life Tri-level last year and had a serve clocked at 122. :shock: He's going to get bumped to 4.0 this year, though, I'd bet.

SlapShot, are you W.C. or P.N., and your partner P.J.? I play 3.5, too, and figure I must know you guys. I'm Doug Rogers.
 
Last edited:
SlapShot, are you W.C., and your partner P.J.? I play 3.5, too, and figure I must know you guys. I'm Doug Rogers.

My partner is in fact P.J. He usually plays with W.C., but he and I have played a few tourneys here and there. I'm P.N. - I play on his 3.5 team.

Edit: W.C. and P.J. are both getting bumped to 4.0 if the current ratings stand.
 
Getting Rated too High

Lot a stories here about people rated too low but none on getting rated too high.

I started playing tennis after a 20 year layoff. Even when I was a kid I didn't play seriously, just banged the ball around. No stroke fundamentals...nada. So I start playing again and 2 months later get talked into playing league because they needed bodies. I self-rate at 3.0 but have to play on a 3.5 doubles league because there weren't enough people to make up a 3.0 team.

Next thing I know I'm paired with a really strong 3.5 who's ends up carrying me throughout the season. Realistically, he could have won all on his own with me sitting on the bench other than to serve (mostly double faults). Anyway, to make a long story short(er)...end of year ratings come out and I'm suddenly bumped up to 3.5 but can't really compete at the 3.5 level unless I had a super strong partner. In singles I would get beat 6-2 or 6-1 against a 3.5 player. Against a 3.0, I'd win some, lose some but it was usually pretty competitive.

I see this a lot, where people play up, get paired with a strong doubles partner, then end up getting moved up at the end of the season when they realistically shouldn't.
 
Lot a stories here about people rated too low but none on getting rated too high.

There is one kid in Arkansas who got screwed by the ratings he self-rated at 4.0 and lost once at Sectionals, was in HS when he started, but this fall is playing d2 tennis, he got rated 5.0, he should be 4.5. I expected him to get bumped but a double-bump is not fair.
 
But if he is playing D2 he will quickly get to 5.0 if he is not already since he will be playing alot with coaching. Young players can continue to improve rapidly while most adults don't improve as quickly.
 
Lot a stories here about people rated too low but none on getting rated too high.

Yeah, good point. In perusing the 2008 ratings in Norcal of people I have played against and/or know it is staggering and amusing to see some of the people that got moved up a level. Out of several dozen players I saw only one person that had their rating go down. It is also surprising considering that Norcal teams only had one win at the Nationals this year. Unless a lot people appeal their rating or Norcal makes some adjustments in the next few days I am not looking forward to next year's mens season as the quality of tennis is likely going to be very inconsistent. A few years ago there was a trend to move ratings down however in the past two years there has been a large number of players moving from 3.0 to 3.5 and 3.5 to 4.0. USTA National must think there are a heck of lot really good players in Norcal if the ratings are any reflection.
 
Yeah, good point. In perusing the 2008 ratings in Norcal of people I have played against and/or know it is staggering and amusing to see some of the people that got moved up a level. Out of several dozen players I saw only one person that had their rating go down. It is also surprising considering that Norcal teams only had one win at the Nationals this year. Unless a lot people appeal their rating or Norcal makes some adjustments in the next few days I am not looking forward to next year's mens season as the quality of tennis is likely going to be very inconsistent. A few years ago there was a trend to move ratings down however in the past two years there has been a large number of players moving from 3.0 to 3.5 and 3.5 to 4.0. USTA National must think there are a heck of lot really good players in Norcal if the ratings are any reflection.

I think the USTA is correcting the trend of people playing down for the past few years. I noticed that there are a lot of 3.5 players moving to 4.0 in the Northern 3.5 section this year, more than I saw last year. Either the play is getting better (which is possible, but not what I'd say is probable), or they are trying to combat the sandbagging.

The playing down seems to have been a ripple from Norcal and other sections getting known for playing down, so that other sections might stand a chance at Nationals.

My team captain is getting bumped to 4.0, and he already appealed and was denied. I don't know if they've gotten more strict, but from what I've heard, the USTA is starting to try to shift things up a little bit.
 
Yep... you better believe there was a huge outcry within the Pacific Northwest Section when everybody saw that 4.5 rating come out!

The Joelson family is very well known within the section. Aside from all of the accomplishments that I listed in my first post, Joelson's son Brett was the #1 ranked Open player in the section last year and his daughter Ashley plays for UCLA. Brian's brother Brad also plays a lot of tournaments in the Northwest and also had an Open ranking.

I know that there were hundreds of people that went "WHAAAAAAAATTTT?" when Brian got listed as a 4.5, and the USTA was nailed with e-mails. Thankfully for them, they made the appropriate change very quickly. The question that the USTA needs to answer is how the computer could have possibly generated a 4.5 rating for him in the first place?

(By the way, Joelson's good friend that was also on the '03 5.5 Nationals team is still listed as a 4.5, so I'm going to start gathering info on him next. I know his friend was ranked #1 in the PNW Open doubles rankings twice, but he doesn't have any former ATP ranking or Grand Slam experience, so his case is a lot less outrageous.)

Basically what Nationals needs to do is if a former pro is very active in tennis without injuries or being overweight, they need to be set to benchmark. Otherwise, the algorithm will keep bringing them down.

People who are able to challenge these players, should be moved up.

Anyway, 4.5 right now can really be out of level with a lot of 5.0, 5.5, and 6.0 players in the league.

By setting the former pros to at least 5.5, it should help bring people's levels back to the correct levels.
 
I do see more players moving up and down this year so I think things are getting better but still seems like anyone can appeal if they want.
 
At least two of the four matches played may have been against National benchmark players. If he beat them 0 or 1, that would explain how he got bumped up so fast.

Ahah. I'd never seen the back screen. It makes more sense to the computer at least.

He beat one guy 1,3 who bumped to 4.0 (though I beat the guy 10-0 in a proset match so either he had 2 really off days, *shrug*)
Second he beat 2,4, also bumped to 4.0
Third guy 0,1, guy actually bumped down to 3.0 since he won no more than 4 games in any of his 5 matches (most were 0,1, 1,1)
Fourth guy he beat 3,3 was a 3.5 benchmark player.

At least it all makes sense now.
 
RE: When ratings go bad

We had a pretty bizarre bump-up at our club. A 3.0 lady at our played for a 3.5 team. She only played two doubles matches the entire season. Both matches were against comparable teams. Not only did she lose both of those matches, but only won 5 games in the process.
 
Ahah. I'd never seen the back screen. It makes more sense to the computer at least.

He beat one guy 1,3 who bumped to 4.0 (though I beat the guy 10-0 in a proset match so either he had 2 really off days, *shrug*)
Second he beat 2,4, also bumped to 4.0
Third guy 0,1, guy actually bumped down to 3.0 since he won no more than 4 games in any of his 5 matches (most were 0,1, 1,1)
Fourth guy he beat 3,3 was a 3.5 benchmark player.

At least it all makes sense now.

Yup ... basically the first two matches set him to 4.0, and the fourth match sealed his fate since it was against a benchmark 3.5 player.

On the first match, the former 3.5 player could have been trying to manipulate his rating, without realizing he was playing against a self-rate :twisted:
 
The system is strange. In Middle States I have a player that did not get more than 2 games in any set, singles or doubles, yet get moved up. This same player was the only person a second player beat, and the second player was move up to a level higher also. Go figure.
 
The system is strange. In Middle States I have a player that did not get more than 2 games in any set, singles or doubles, yet get moved up. This same player was the only person a second player beat, and the second player was move up to a level higher also. Go figure.

I need a little bit more data to figure out what's going on.

For example:

Let's say you self-rate 4.5 and you can only play 5.0 since there are no 4.5 around. You first match, you play against a high 5.0 and lose 0-6, 0-6. That first match will put you at the bottom of 5.0, and not push you down to a 4.5 as you would expect.

So everything depends on who you play against, even as a computer rated player. Now, if you have 10 years of data saying you are 3.0, and suddenly get moved to 4.0 after a poor season, that would be strange indeed.
 
Here's a good one: http://tinyurl.com/2dcofj

5.0 guy with 4 years of matches in the computer plays 10.0 mixed, goes 9-1, including matches at the state and sectional tournaments.

Based on this record, he gets dropped 2 levels, to 4.0, by the almighty computer.


Yes, he also played 9.5 combo, but in the Southern section that doesn't count towards your rating. A policy which confuses me greatly: surely, if the computer can come up with meaningful ranking for mixed doubles results, it can surely do so for combo doubles.
 
Here's a good one: http://tinyurl.com/2dcofj

5.0 guy with 4 years of matches in the computer plays 10.0 mixed, goes 9-1, including matches at the state and sectional tournaments.

Based on this record, he gets dropped 2 levels, to 4.0, by the almighty computer.


Yes, he also played 9.5 combo, but in the Southern section that doesn't count towards your rating. A policy which confuses me greatly: surely, if the computer can come up with meaningful ranking for mixed doubles results, it can surely do so for combo doubles.

How can an accurate algorithm even come up with this crap? He was winning at 10.0 mixed and he hardly lost at 5.0 in 2006? This has to be a joke!!!
Hopefully this gets nipped real quick by Southern, contact Phil Meadors the head of Southern Leagues.
 
A socal team went to 5.0 nationals (made it to semis) in 2005.
Several players (who won almost all their matches), appeal down
and play 4.5 in 2007.

Doesn't seem that bad compared to some of the other examples
listed. Ultimately it's all for fun and a chance to play so probably not
that big of a deal unless you take it really seriously.
 
Back
Top