PDA

View Full Version : Post the most absurd 2010 NTRP rating examples - no names!


ian2
12-01-2010, 09:14 AM
It's that time of the year again! No I don't mean the holidays. The 2010 NTRP ratings are out. Post your favorite examples of the most absurd rating outcomes you've come upon. No names/links/locations please! Just the stats. My hope is that the examples would shed light on some of the ways in which NTRP algorithm is broken. BTW, if you believe that it is NOT broken and want to argue this point, let's take that argument to another thread. I'd like to ask everyone to just post examples here, no opinions, and most certainly no info that could reveal the actual names behind the examples!

JoelDali
12-01-2010, 10:23 AM
Deleted ... Durrr ...

ian2
12-01-2010, 10:32 AM
Thanks Senor Dali... nice limerick in the other tread BTW.

Anyone else?

ian2
12-01-2010, 10:46 AM
OK a couple of examples I've run across...

Example 1:

2009 4.5C player match results:

4.5 singles: 26:3 win/loss. The three losses were in close matches to players who has similarly dominant records.

4.5 doubles: 1:0 win/loss.

Did not play any other matches.

2010 NTRP rating: unchanged, 4.5C
__________________

Example 2:

2009 3.5C player match results:

3.5 singles: 2 wins/6 losses. Both wins were competitive scores against the same 3.5 player with 0:9 win/loss ratio for the year. Two of the losses were 0 and 2, 0 and 0; remaining losses were competitive.

3.5 doubles: 8 wins/1 losses. The loss was 1 and 5. Best win was 0 and 2; the rest of the wins were competitive.

4.0 doubles: 0 wins/8 losses. Best loss was in split sets; worst loss 1 and 0; the rest of the losses averaged 2 and 3.

2010 NTRP rating: bumped up to 4.0C

robert
12-01-2010, 10:59 AM
Have a guy played two 4.0 single matches in whole season. Lost both with double bageled. And he is still remain 4.0C. How come?

ian2
12-01-2010, 11:12 AM
Have a guy played two 4.0 single matches in whole season. Lost both with double bageled. And he is still remain 4.0C. How come?
This one is no mystery actually. 6:0 6:0 scores are thrown away because (get this!) DNTRP algorithm cannot (or would not) make a distinction between a default recorded as 0 and 0 and an actual match that was played.

obtn
12-01-2010, 11:13 AM
Have a guy played two 4.0 single matches in whole season. Lost both with double bageled. And he is still remain 4.0C. How come?

I have hard on other threads that bagels don't count towards ratings at all.

The most I can contribute to this thread is that the number 1 ranked 3.5 guy in my state is still a 3.5 (A), dated 12/31/2009... not sure if he just didn't get updated or what.

Jim A
12-01-2010, 11:15 AM
I think having an algorithm is fine because if it was up directly to Section Coordinators, etc then it would be even more of a mess.

A rating is a starting point to put together fairly competitive groupings on paper. Once you get on the court, it can all go to pot. We see it week in and week out, I can beat Player A but not Player B, however Player A owns Player B when they play.

You can cheat in any sport with enough thought. Some players will be bumped up too early, others held back for what appears to be no rhyme or reason. I've spoken about a couple people in my area who I have been surprised the past 2 years stayed at level.

If you lose 2 and 3 I wouldn't expect to be bumped down. Is David Ferrer getting dropped out of the ATP tour for losing the way he did last week?

The player you reference @ 3.5 appears to be more suited at dubs, so he won out at 3.5 dubs but lost most matches on average 5-6 games won @ 4.0 ....so he should go back to 3.5 where he can win out again? It would appear you lost a teammate

ian2
12-01-2010, 11:23 AM
I have hard on other threads that bagels don't count towards ratings at all.

The most I can contribute to this thread is that the number 1 ranked 3.5 guy in my state is still a 3.5 (A), dated 12/31/2009... not sure if he just didn't get updated or what.

#1 ranked 3.5 guy in my state got a new ranking... same as the old one, 3.5C. Ditto for #1 ranked 3.5 dubs team... they won all there was to win, and stayed 3.5.

njsigman
12-01-2010, 11:25 AM
I thought these results were absurd, but based on the other thread about playing up, it might answer some questions. This guy I know got bumped from 3.0 to 3.5.

0-8 combined on the season between 3.0 and 3.5 with these results

Score Match Type Level
6-4, 4-6, 4-3 #2 Doubles 3

6-0, 6-3 #3 Doubles 3.5

6-0, 6-1 #1 Doubles 3.5

6-2, 6-1 #1 Doubles 3

4-6, 6-3, 3-2 #2 Doubles 3.5

6-0, 6-1 #2 Doubles 3.5

7-6, 5-3 #3 Doubles 3.5

6-4, 6-2 #2 Doubles 3.5

ian2
12-01-2010, 11:37 AM
I think having an algorithm is fine because if it was up directly to Section Coordinators, etc then it would be even more of a mess.

A rating is a starting point to put together fairly competitive groupings on paper. Once you get on the court, it can all go to pot. We see it week in and week out, I can beat Player A but not Player B, however Player A owns Player B when they play.

You can cheat in any sport with enough thought. Some players will be bumped up too early, others held back for what appears to be no rhyme or reason. I've spoken about a couple people in my area who I have been surprised the past 2 years stayed at level.

Jim, having algorithm is fine indeed... except the fact that algorithm produces nonsensical results way too often. It's not about actual ability or match-ups, no algorithm could handle that. It's about the fact that given hard cold numbers, the algorithm generates garbage outcomes. Again, not always but in my opinion way too frequently.

If you lose 2 and 3 I wouldn't expect to be bumped down. Is David Ferrer getting dropped out of the ATP tour for losing the way he did last week?

Sorry, I lost you here. What do you mean?

The player you reference @ 3.5 appears to be more suited at dubs, so he won out at 3.5 dubs but lost most matches on average 5-6 games won @ 4.0 ....so he should go back to 3.5 where he can win out again? It would appear you lost a teammate
There isn't a separate rating for dubs and singles. The guy's performance in 3.5 was a wash at best yet he was bumped up, unlike the guy in #1 example I gave. Compare their records and tell me whether it makes any sense... Neither guy is my teammate BTW although I do know them both.

J_R_B
12-01-2010, 12:00 PM
I thought these results were absurd, but based on the other thread about playing up, it might answer some questions. This guy I know got bumped from 3.0 to 3.5.

0-8 combined on the season between 3.0 and 3.5 with these results

Score Match Type Level
6-4, 4-6, 4-3 #2 Doubles 3

6-0, 6-3 #3 Doubles 3.5

6-0, 6-1 #1 Doubles 3.5

6-2, 6-1 #1 Doubles 3

4-6, 6-3, 3-2 #2 Doubles 3.5

6-0, 6-1 #2 Doubles 3.5

7-6, 5-3 #3 Doubles 3.5

6-4, 6-2 #2 Doubles 3.5

This is no mystery. He played several competitive matches at 3.5. Unless all of those were against bottom of the barrel 3.5 players, he is going to get bumped.

J_R_B
12-01-2010, 12:02 PM
Jim, having algorithm is fine indeed... except the fact that algorithm produces nonsensical results way too often. It's not about actual ability or match-ups, no algorithm could handle that. It's about the fact that given hard cold numbers, the algorithm generates garbage outcomes. Again, not always but in my opinion way too frequently.



Sorry, I lost you here. What do you mean?


There isn't a separate rating for dubs and singles. The guy's performance in 3.5 was a wash at best yet he was bumped up, unlike the guy in #1 example I gave. Compare their records and tell me whether it makes any sense... Neither guy is my teammate BTW although I do know them both.

What is "way too frequently"? The algorithm processes thousands and thousands of results and gets it right 95% or more. Can it be improved? Probably, but it's still not doing poorly over all.

Jim A
12-01-2010, 12:08 PM
Ian, I think you and I are in the same section and general level of play.

If you are looking at tournament rankings, they don't match the league play. The top singles players in my district all moved up. there are some good ones who didn't and were likely on the bubble. I think in general the level in my area appears to be a bit below the national standard. The players who are winning still aren't at the next level.

Some of the players who have a lot of points from the tournament side aren't all that good. They just play every week and when there's a weak draw...find themselves in the semi's finals.

I used Ferrer as an example because he is the #8 ranked player in the world despite having weak results at the GS. He got thumped in the WTF against some of the top players, however would you say he isn't "competitive". Rating levels are much the same.

as for singles and doubles, since it combines it all together he went 10-7 last year at 3.5. with only 2 bad losses per the use of competitive being ~ 4 games..so 15/17matches were either competitive or won...when the computer looks at that how do you think it is viewed?

magmasilk
12-01-2010, 12:14 PM
I think the esential missing piece in "reverse engineering" the ratings algorithm is the "computer rating/strength" of the opponent(s). Line number is a poor proxy.

Additionally, pretty sure Win-Loss doesn't directly matter (only games won). But not sure how many historical matches matches are considered. I would venture it would potentially looks back at more than a calander years worth of matches.

Overall, when I looked at a few rating bumps it seemed pretty consistent in that if you beat or played close someone who was bumped you most likely got bumped.

ian2
12-01-2010, 02:13 PM
I'm not saying that the algorithm is not working in general. It works well in the middle of any given NTRP level. What I'm seeing though is than it starts wobbling at the margins and gets completely insane under certain scenarios. Can anyone seriously argue that the year-end ratings make sense for the two examples I have given? Especially when you look at these two examples side-by-side?

Yes we can explain how the algorithm arrives at these results (to a certain degree, given the fact that the system is intentionally opaque.) My point is, it arrives at some of these quite obviously absurd results because it can't handle certain scenarios. In other words, these are NOT the result intended by the designers of the algorithm; instead these are the product of the bugs in algorithm logic.

This thread was intended as a "catalog" of such bugs. To fix something that is broken it helps to understand how exactly it is broken. Not that USTA is necessarily looking to fix the algorithm problems. It appears that their priority lately had been to manipulate the algorithm application so that it would produce the desired "distribution" - which by the way is a valid approach to addressing some of the issues.

kylebarendrick
12-01-2010, 02:34 PM
I'd say your two examples were both likely borderline. Example 2 is obvious - they played competitive matches while playing up so they got bumped. For example 1 you may be putting too much of an emphasis on W/L results. If the bulk of their wins were against mid-level players (or even some against 4.0s playing up) then I can understand why they weren't bumped.

I think you are right, though. It is at the boundaries between levels where the system struggles - especially with people playing at more than one level. As long as people don't try to manipulate the system, I can accept that it isn't perfect. None of us are going to be playing on the pro tour anytime soon.

ian2
12-01-2010, 02:41 PM
Ian, I think you and I are in the same section and general level of play... I too got that impression, form some of your posts in other threads.

If you are looking at tournament rankings, they don't match the league play. The top singles players in my district all moved up. there are some good ones who didn't and were likely on the bubble. I think in general the level in my area appears to be a bit below the national standard. The players who are winning still aren't at the next level.

Some of the players who have a lot of points from the tournament side aren't all that good. They just play every week and when there's a weak draw...find themselves in the semi's finals.

I used Ferrer as an example because he is the #8 ranked player in the world despite having weak results at the GS. He got thumped in the WTF against some of the top players, however would you say he isn't "competitive". Rating levels are much the same.

Jim, we probably are in the same geographical section and general level of play, as you said... I got that impression from some of your posts in other threads.

I disagree somewhat with your point that "some of the players who have a lot of points from the tournament side aren't all that good." I mean, "that good" is a relative term :-) Sure, some of the best players at a given level do not play tournaments at all. However it's important to remember that only three best tournament results are counted. Bottom line is that top 10 ranked 3.5 players (which is my level though I'm not one of them :cry:) are all pretty darn good as 3.5 goes. And the fact that both #1 singles and #1 doubles guys did not make it to 4.0 illustrates some of the problems I was talking about. Which by the way ties into that Ferrer example: imagine if the pros were rated on their game scores and who they played rather than on the match outcomes? I'm not saying that we amateurs should be ranked similarly to pros. But I am saying that not taking win/loss record into consideration (as one of the parameters, not the only parameter!) produces some of these nonsensical outcomes.

as for singles and doubles, since it combines it all together he went 10-7 last year at 3.5. with only 2 bad losses per the use of competitive being ~ 4 games..so 15/17matches were either competitive or won...when the computer looks at that how do you think it is viewed?
And when computer looks at 27:3 at-level record with half of the wins complete blow-outs and the losses essentially ties, how do you think it is viewed? Again, do you see a modicum of consistency or common sense here? The 10-7 3.5 guy becomes 4.0; the 27-3 4.5 guy remains 4.5?

ian2
12-01-2010, 02:58 PM
I'd say your two examples were both likely borderline. Example 2 is obvious - they played competitive matches while playing up so they got bumped. For example 1 you may be putting too much of an emphasis on W/L results. If the bulk of their wins were against mid-level players (or even some against 4.0s playing up) then I can understand why they weren't bumped.

I think you are right, though. It is at the boundaries between levels where the system struggles - especially with people playing at more than one level. As long as people don't try to manipulate the system, I can accept that it isn't perfect. None of us are going to be playing on the pro tour anytime soon.

kyle, I don't see example 2 as obvious, not at all. The guy was borderline competitive at 4.0 dubs but at the same time he was less than competitive at 3.5 singles. The 3.5 dubs record was OK but not outstanding. Does this "obviously" equal a bump to 4.0? No, I don't think so. Remaining a 3.5 would appear to be a more "obvious" outcome.

The 4.5 guy with 27 - 3 record: half of his wins were from league play, so he obviously played 4.5 players there. Many of his wins were blow-outs. The rest were in tournaments. He might have played some 4.0 players along the way but he sure as hell beat some strong 4.5 players on his way to winning three of these tournaments... which earned him a very high state ranking. Given this info, do you still understand why he wasn't bumped to 5.0? I absolutely don't.

kylebarendrick
12-01-2010, 05:31 PM
When I wrote "obvious", I meant that this is what likely happened. I think that the result that he got bumped up could have gone either way - and that is the risk of playing up. I agree that I'd have like to see the 4.5 get moved up. Do you know that tournaments are included in the ratings in your section? They aren't in all sections and how they are included is even more obscure than league results!

I'm not saying that the computer results for these guys was obvious - I am assuming that the both ended up close to the boundaries and agreed with your statement that the computer struggles with the boundary cases. I would like to see some weight given to W/L records since things like a 26-3 record do scream for a bump regardless of the scores!

Jim A
12-01-2010, 05:36 PM
Ian, here is all I can say about my experience with tournament/league play in my Section.

In '09 I played 3 events and won them all, beat everyone ranked ahead of me. My toughest matches were in league by far, lost more sets and games throughout the season.

I just took a look at the Tournament Rankings at my level. I've seen many of those guys play and while good 3.5's aren't a necessarily 4.0's.

The key to be ranked highly at the end of the year if you are a better than average 3.5 is to play a lot. If you view the results many of them play a ton of events and when the draw breaks their way, they cash in. Only needs to happen 2-3 times/yr and you are Top 10.

One of my teammates lost the majority of his league matches and knocked off one of the top ranked, and he was the better player on the court.

Many of the top league players don't play tournaments or if they do its only 1-2 per year and not enough to make a dent overall, they may only play State Open, etc....

No idea on the 27:3 guy, maybe he played a ton of people bumped this year, but since 5.0 is limited in many areas, I'm sure the computer is going to be certain before pushing someone to the last ledge of USTA play.

Vik
12-01-2010, 05:48 PM
kyle, I don't see example 2 as obvious, not at all. The guy was borderline competitive at 4.0 dubs but at the same time he was less than competitive at 3.5 singles. The 3.5 dubs record was OK but not outstanding. Does this "obviously" equal a bump to 4.0? No, I don't think so. Remaining a 3.5 would appear to be a more "obvious" outcome.

The 4.5 guy with 27 - 3 record: half of his wins were from league play, so he obviously played 4.5 players there. Many of his wins were blow-outs. The rest were in tournaments. He might have played some 4.0 players along the way but he sure as hell beat some strong 4.5 players on his way to winning three of these tournaments... which earned him a very high state ranking. Given this info, do you still understand why he wasn't bumped to 5.0? I absolutely don't.

Um, it's because a crap load of 3.5s were bumped to 4.0. USTA is not as active in bumping 4.5s to 5.0. That's really the main flaw in what they've done the last 2 years. They didn't build up the 5.0 ranks. 4.5 is way diluted.

beststringer
12-01-2010, 07:02 PM
Someone 4.0 C, 0-5 record, got bumped to 4.5

dlk
12-02-2010, 04:22 AM
I know that alot of people inflate their ability; with that said, I was typically the best player on the court when playing doubles. I could double bagel my partner 90% of the time.


I was a 3.0S (1st year in USTA) 2 total years tennis experience. Better than average athletic ability.

Mens3.5 record= 1-3
Mixed 6.5= 1-5
Mixed 6.0= 2-6

I was bumped up, and I'm a little confused, I wanted to play Mens 3.0, to see what I could do. I do think I could be competitive at 3.5, as was last year, and continue to improve.

Totai
12-02-2010, 06:19 AM
I only played 5 mixed doubles matches this year for USTA, won 2 easily, 2 competitively, and lost 1, and I got bumped up to 4.0 -_-

gameboy
12-02-2010, 07:46 AM
Mens3.5 record= 1-3
Mixed 6.5= 1-5
Mixed 6.0= 2-6

I was bumped up, and I'm a little confused, I wanted to play Mens 3.0, to see what I could do. I do think I could be competitive at 3.5, as was last year, and continue to improve.

I am confused that you are confused.

Why would you play up at 3.5 if you wanted to play 3.0??? The fact that you won a match at 3.5 says that you are 3.5. Why are you confused?

I still have yet to see any egregious rating examples. Most are just hyperventilating with no reason. If you have competitive (and THAT is the key word - COMPETITIVE - it doesn't matter if you lose, you just have to be competitive to get ranked up) matches at the higher level, that is where you are going to be ranked, it does not matter if you have a comparably horrible record in a lower level.

jdubbs
12-02-2010, 08:01 AM
I'm having fun at the 4.0 level, though I'm 0-2 in first round tournament matches so far. Both matches were competitive, though, and I'm sure I'll break through at some point ha ha.

But looking at the winner of the last 4.0 tournament I played in, he played 9 matches in 2 tournaments, won them both, and only dropped 1 set.

So I don't have any illusions that I probably won't be winning any tourneys, but it's fun anyway and I've gotten a lot better in the last few months since i started focusing on match play.

cknobman
12-02-2010, 08:03 AM
I have a document (pre 2009) that goes into great detail on how the USTA dynamic rating system calculates your rating. Yes this is an official USTA document that was leaked to me from a friend.

Anytime you play up a level your dynamic rating is going to go up regardless of whether you win or loose (unelss you get double bageled) because your playing against players who are rated higher than you.

On the same note players playing against someone who is playing up are almost certainly have their rating go down whether they win or loose as well, because they are playing against lower rated players.

Most of the time when you see something whacky happen to a player when it comes to ratings you should look closer than just their flat out record of wins/losses. Look at who they played and what their rating was when they played that person.

A person could go 8-0 in a season but if every win they had was against players playing up a level then you will likely see that person get bumped down.

Likewise a person could go 0-8 and get bumped up if most of their matches were a level higher than what they are rated. Its simply that person plays higher rating makes that persons rating go up.

If you want an accurate rating then you need to make sure you play players rated the same as yourself, simple as that.

ian2
12-02-2010, 08:42 AM
I have a document (pre 2009) that goes into great detail on how the USTA dynamic rating system calculates your rating. Yes this is an official USTA document that was leaked to me from a friend.

Anytime you play up a level your dynamic rating is going to go up regardless of whether you win or loose (unelss you get double bageled) because your playing against players who are rated higher than you.

On the same note players playing against someone who is playing up are almost certainly have their rating go down whether they win or loose as well, because they are playing against lower rated players.

Most of the time when you see something whacky happen to a player when it comes to ratings you should look closer than just their flat out record of wins/losses. Look at who they played and what their rating was when they played that person.

A person could go 8-0 in a season but if every win they had was against players playing up a level then you will likely see that person get bumped down.

Likewise a person could go 0-8 and get bumped up if most of their matches were a level higher than what they are rated. Its simply that person plays higher rating makes that persons rating go up.

If you want an accurate rating then you need to make sure you play players rated the same as yourself, simple as that.

I'd love to take a look at that document... could you possibly email it to me privately?

I absolutely agree with your statement (in bold). A different way to phrase it is "If you want an inflated rating you need to play up". Yet another way to express the same idea is "You performance playing at-level is disproportionately devalued as compared to performance of someone playing up".

People choose to play up, for a variety of reasons. There is nothing wrong with that. The wrong thing is that the algorithm cannot handle these situations. It demonstrably breaks down under such conditions and produces results that range from borderline "reasonable" to patently absurd.

gameboy
12-02-2010, 08:50 AM
People choose to play up, for a variety of reasons. There is nothing wrong with that. The wrong thing is that the algorithm cannot handle these situations. It demonstrably breaks down under such conditions and produces results that range from borderline "reasonable" to patently absurd.

What is wrong with the algorithm? If the player plays up and is competitive, then that person should play in that level. It is pretty simple. How does it "demonstrably breaks down"?

ian2
12-02-2010, 09:06 AM
Ian, here is all I can say about my experience with tournament/league play in my Section.

In '09 I played 3 events and won them all, beat everyone ranked ahead of me. My toughest matches were in league by far, lost more sets and games throughout the season.

I just took a look at the Tournament Rankings at my level. I've seen many of those guys play and while good 3.5's aren't a necessarily 4.0's.

The key to be ranked highly at the end of the year if you are a better than average 3.5 is to play a lot. If you view the results many of them play a ton of events and when the draw breaks their way, they cash in. Only needs to happen 2-3 times/yr and you are Top 10.

One of my teammates lost the majority of his league matches and knocked off one of the top ranked, and he was the better player on the court.

Many of the top league players don't play tournaments or if they do its only 1-2 per year and not enough to make a dent overall, they may only play State Open, etc....

No idea on the 27:3 guy, maybe he played a ton of people bumped this year, but since 5.0 is limited in many areas, I'm sure the computer is going to be certain before pushing someone to the last ledge of USTA play.

Jim, my experience playing league and tournaments this year was the opposite. Relatively easy going in leagues, much tougher competition in tournaments... could be just the lack of the draw. I still stand by my statement that highly ranked tournament players (top 10 for sure) represent the top range of a given NTRP level, regardless of the fact that some of the top players at a level do not play tournaments or play only a couple all year. As to the question of whether a top-10 3.5 is "4.0 material" - yes, absolutely, in the sense that these top 3.5s will beat bottom 25% of 4.0s 90% of the time and will beat lower 50% of 4.0s half of the time. Sorry about all the fractions :-)

dlk
12-02-2010, 09:20 AM
I am confused that you are confused.

Why would you play up at 3.5 if you wanted to play 3.0??? The fact that you won a match at 3.5 says that you are 3.5. Why are you confused?

I still have yet to see any egregious rating examples. Most are just hyperventilating with no reason. If you have competitive (and THAT is the key word - COMPETITIVE - it doesn't matter if you lose, you just have to be competitive to get ranked up) matches at the higher level, that is where you are going to be ranked, it does not matter if you have a comparably horrible record in a lower level.

It was my first year playing USTA, didn't (and probably still don't) understand ratings. Played against a guy in WTT, who asked me if I wanted to play on his team; he said "you can always play up." I did not understand that this would bump me up. Thanks for the info, I believe I have a better understanding now.

ian2
12-02-2010, 09:22 AM
What is wrong with the algorithm? If the player plays up and is competitive, then that person should play in that level. It is pretty simple. How does it "demonstrably breaks down"?

Please take another look at the two examples I listed at the top of this thread. The only way you can interpret such outcomes is that the algorithm gives disproportionately higher weight to the "play-up" results. A players who is barely competitive at his own level gets bumped up due to simply playing up (losing all matches at the higher level, and losing convincingly except in one match). A player dominating his own level stays put. Also take a look at some of the examples other posters contributed. If such outcomes were "as designed", the design is flawed in my opinion. If they were not "as designed", the algorithm broke down...

OrangePower
12-02-2010, 09:33 AM
This one is no mystery actually. 6:0 6:0 scores are thrown away because (get this!) DNTRP algorithm cannot (or would not) make a distinction between a default recorded as 0 and 0 and an actual match that was played.

0 & 0 scores do get thrown out, but not for the reason you suggest - defaults are indicated as such and it would be easy to exclude them from DNTRP consideration.

The reason the algorithm can't evaluate 0 & 0 scores is that it is the most lopsided score, and so the gap between the 2 players could range from very large to stupendously large, with no way to distinguish. For example, if I lose 0 and 0 to a pro, you could conclude that I'm no better than a 6.5 maybe, but you'd (and the rating algorithm) would have no way to deduce whether I'm a 2.5 or a 6.0.

People choose to play up, for a variety of reasons. There is nothing wrong with that. The wrong thing is that the algorithm cannot handle these situations. It demonstrably breaks down under such conditions and produces results that range from borderline "reasonable" to patently absurd.

The algorithm doesn't do well in these cases partly because of similar reasons to the 0 and 0 situation above. Another reason is that the algorithm relies on actual match scores, but actual match scores are not always a true indicator of difference in ability between players (but is all the algorithm has to go on).

For example, a 6-1 6-2 score is what we would expect if say a top 4.0 were to play a mid to low level 4.0. Now let's say the same top level 4.0 plays a top-level 3.5. In this case, we'd expect 0 & 0, or maybe 1 & 0. And if the top 4.0 is really trying, that's what we'd get.

But in reality, it's hard to keep that level of focus when you're clearly superior and have the match in hand. And also, the weaker player has nothing to lose and maybe 'goes for it' and has some luck. So you can see how the score could easily end up as 1 & 2... same as it would be for the top 4.0 versus the other 4.0. And the algorithm obviously can't distinguish.

Elisa
12-02-2010, 09:53 AM
The only way you can interpret such outcomes is that the algorithm gives disproportionately higher weight to the "play-up" results.

There are several other ways to interpret such outcomes.

USTA weighs each match the same in terms of your dynamic rating. Every player has a dynamic rating and that rating changes- either a lot or a little- following every match.

Singles, doubles, playing in your level, playing up a level, etc. are all weighed the same. However, when you play singles or someone with a rating that is much higher or much lower than yours, your rating has the potential for greater change.

Let's use the example of Bill, an average 3.0 player. Bill starts the season with a 2.75 dynamic rating. Bill plays one of these 4 different matches with different opponents:

Bill d. Steve (3.0 with a 2.65 dynamic rating) 6-4 6-4

The computer expects close to this score, the match generates a 2.78 rating for Bill for this match, which gets averaged into his dynamic rating. Bill's dynamic rating goes up to 2.76

Albert (3.5 with a 3.45 dynamic rating) d. Bill 6-4 6-3

The computer expects a bigger differential based on their ratings, so Bill's rating for this match is 3.22. This gets averaged into Bill's dynamic rating, Bill has a new dynamic rating of 2.89. Albert gets a new dynamic rating of 3.40.

When you play doubles, essentially the change in the rating applied to the players is divided by 4 instead of dividing by 2. This means that you can change your rating more and get bumped faster in singles, but not that singles is weighted more heavily.

aniretake
12-02-2010, 10:06 AM
I played last year, went to state (won half of matches, the other half lost in tie-breakers). Was a finalist (one division) and a winner (other division) in a local Banana Open. Played all Fall at 3.5 and low 4.0 leagues- did well. And I got bumped down to 3.0!!! :(
My former team-mate played only for a year, did not go to state with us, then quit playing completely and got bumped up to 3.5.:shock:

ian2
12-02-2010, 10:14 AM
0 & 0 scores do get thrown out, but not for the reason you suggest - defaults are indicated as such and it would be easy to exclude them from DNTRP consideration.

The reason the algorithm can't evaluate 0 & 0 scores is that it is the most lopsided score, and so the gap between the 2 players could range from very large to stupendously large, with no way to distinguish. For example, if I lose 0 and 0 to a pro, you could conclude that I'm no better than a 6.5 maybe, but you'd (and the rating algorithm) would have no way to deduce whether I'm a 2.5 or a 6.0.

Yes I see what you are saying about the gap potentially ranging from "very large to stupendously large". However the algorithm "knows" the DNTRP of each player beforehand. Why then the 6:0 6:0 scores between two 3.5 players are thrown out? Is it because there might "stupendously large" gap in ability? Nope. These scores are thrown out because throwing out ALL 6:0 6:0 scores is an easy thing to do. Lazy algorithm design basically.

The algorithm doesn't do well in ... play-up <commented by Ian2> ...cases partly because of similar reasons to the 0 and 0 situation above. Another reason is that the algorithm relies on actual match scores, but actual match scores are not always a true indicator of difference in ability between players (but is all the algorithm has to go on).

For example, a 6-1 6-2 score is what we would expect if say a top 4.0 were to play a mid to low level 4.0. Now let's say the same top level 4.0 plays a top-level 3.5. In this case, we'd expect 0 & 0, or maybe 1 & 0. And if the top 4.0 is really trying, that's what we'd get.

But in reality, it's hard to keep that level of focus when you're clearly superior and have the match in hand. And also, the weaker player has nothing to lose and maybe 'goes for it' and has some luck. So you can see how the score could easily end up as 1 & 2... same as it would be for the top 4.0 versus the other 4.0. And the algorithm obviously can't distinguish.

Exactly, it cannot, and that's the problem. But it could, at least to a certain degree. It attempts to do this already, through the concept of "competitive score" (admittedly I don't know the exact details since the algorithm is kept secret). My feeling is that it could be "calibrated" much better for the "play up" scenarios, at least to the point of eliminating the most absurd outcomes.

ian2
12-02-2010, 10:32 AM
There are several other ways to interpret such outcomes.

USTA weighs each match the same in terms of your dynamic rating. Every player has a dynamic rating and that rating changes- either a lot or a little- following every match.

Singles, doubles, playing in your level, playing up a level, etc. are all weighed the same. However, when you play singles or someone with a rating that is much higher or much lower than yours, your rating has the potential for greater change.

Let's use the example of Bill, an average 3.0 player. Bill starts the season with a 2.75 dynamic rating. Bill plays one of these 4 different matches with different opponents:

Bill d. Steve (3.0 with a 2.65 dynamic rating) 6-4 6-4

The computer expects close to this score, the match generates a 2.78 rating for Bill for this match, which gets averaged into his dynamic rating. Bill's dynamic rating goes up to 2.76

Albert (3.5 with a 3.45 dynamic rating) d. Bill 6-4 6-3

The computer expects a bigger differential based on their ratings, so Bill's rating for this match is 3.22. This gets averaged into Bill's dynamic rating, Bill has a new dynamic rating of 2.89. Albert gets a new dynamic rating of 3.40.

When you play doubles, essentially the change in the rating applied to the players is divided by 4 instead of dividing by 2. This means that you can change your rating more and get bumped faster in singles, but not that singles is weighted more heavily.

Elisa, I understand that this is how the system works (a very good walk-through BTW). What I'm saying is that the way it works generates absurd results on the edges, i.e. in play-up situations. This happens, it appears, because the weighting is "off". The large differential in DNTRP between the two players results in the lower-rated player getting a disproportionally high DNTRP increase and the higher-rated player a disproportionally high DNTRP decrease. "Disproportionally high" as compared to the same scores between two players with closer "starting" DNTRP. A matter of calibration essentially. This compounds and in the end causes these strange outcomes. As I said a number of times through this thread: please take a look at the two examples I've given in the beginning. Yes these outcomes can be explained away. But do they make sense?

cknobman
12-02-2010, 10:44 AM
I'd love to take a look at that document... could you possibly email it to me privately?

I absolutely agree with your statement (in bold). A different way to phrase it is "If you want an inflated rating you need to play up". Yet another way to express the same idea is "You performance playing at-level is disproportionately devalued as compared to performance of someone playing up".

People choose to play up, for a variety of reasons. There is nothing wrong with that. The wrong thing is that the algorithm cannot handle these situations. It demonstrably breaks down under such conditions and produces results that range from borderline "reasonable" to patently absurd.

Sure I can email it to you when I get home from work. Keep in mind though this is before they changed the algorithm last year so I dont know how much of it is still valid but the document outlines how it was done pre eoy 2009.

ian2
12-02-2010, 10:49 AM
Sure I can email it to you when I get home from work. Keep in mind though this is before they changed the algorithm last year so I dont know how much of it is still valid but the document outlines how it was done pre eoy 2009.
Awesome, thanks!

Jim A
12-02-2010, 05:01 PM
We have very different experiences by far Ian2, interesting how it changes per person. Luckily we have the internet. Maybe we'll wind up facing one another next year!

In order to protect the innocent, I'll say that some of those top ranked players won only 3/7 of their tournament matches and about others hovered just a bit above .500. Quantity not quality, just review the matches on Tennislink and see who the wins/losses were against.

My teammate was 8-17 on the year and took one of them down and as stated earlier, he was the better player by far and it wasn't a fluke

The thing with 3.5 out here is the guy you play in the 1st round likely isn't that much different than the one who wins the tournament. There's a lot of parity once you get past the true beasts who have all moved up (and pretty much stayed to league play). If there are 20 people in the draw, 12-14 could realistically walk away with the beer mug

An average 4.0 is way different than some of the people we are discussing, especially in singles. When I play some of my buddies, its 6-3 on a good day and I have to really go for winners to stay in points, which is no way to play 4.0. They are more consistent, have better patterns and weapons. It takes a lot more to be successful from the move from 3.5 to 4.0 then from 3.0 to 3.5.

The people who played up in Twilight before the bump pretty much got rolled...so you definitely cannot just show up at 4.0 and win 50% of your matches because you were good at 3.5

Nellie
12-02-2010, 06:22 PM
Well, I have not played USTA in a couple of years, but still got bumped up last year. I was checking to see If I could have gotten bumped up again! At least this year, my rating did not change.

gameboy
12-02-2010, 10:34 PM
Please take another look at the two examples I listed at the top of this thread. The only way you can interpret such outcomes is that the algorithm gives disproportionately higher weight to the "play-up" results. A players who is barely competitive at his own level gets bumped up due to simply playing up (losing all matches at the higher level, and losing convincingly except in one match). A player dominating his own level stays put. Also take a look at some of the examples other posters contributed. If such outcomes were "as designed", the design is flawed in my opinion. If they were not "as designed", the algorithm broke down...

I've got a news for you.

This is how every rating systems work! Golf, chess, even video games, you name it and they all weigh your best results more than your poorer ones in order to minimize the "gaming" of the system.

There is absolutely nothing "weird" about the examples you have provided.

JoelDali
12-03-2010, 05:54 AM
USTA League ..............

http://www.wildernessvagabonds.com/zp/uploaded/Serious-Business.jpg

ian2
12-03-2010, 07:09 AM
I've got a news for you.

This is how every rating systems work! Golf, chess, even video games, you name it and they all weigh your best results more than your poorer ones in order to minimize the "gaming" of the system.

There is absolutely nothing "weird" about the examples you have provided.

A 3.5 guy gets killed in 3.5 singles, loses every match at 4.0 dubs (most of them badly), and is bumped to 4.0. A 4.5 guy absolutely kills it at 4.5 and is not bumped. Nothing weird about this, sure!

ian2
12-03-2010, 07:39 AM
We have very different experiences by far Ian2, interesting how it changes per person. Luckily we have the internet. Maybe we'll wind up facing one another next year!

In order to protect the innocent, I'll say that some of those top ranked players won only 3/7 of their tournament matches and about others hovered just a bit above .500. Quantity not quality, just review the matches on Tennislink and see who the wins/losses were against.

My teammate was 8-17 on the year and took one of them down and as stated earlier, he was the better player by far and it wasn't a fluke

The thing with 3.5 out here is the guy you play in the 1st round likely isn't that much different than the one who wins the tournament. There's a lot of parity once you get past the true beasts who have all moved up (and pretty much stayed to league play). If there are 20 people in the draw, 12-14 could realistically walk away with the beer mug

An average 4.0 is way different than some of the people we are discussing, especially in singles. When I play some of my buddies, its 6-3 on a good day and I have to really go for winners to stay in points, which is no way to play 4.0. They are more consistent, have better patterns and weapons. It takes a lot more to be successful from the move from 3.5 to 4.0 then from 3.0 to 3.5.

The people who played up in Twilight before the bump pretty much got rolled...so you definitely cannot just show up at 4.0 and win 50% of your matches because you were good at 3.5

Jim, if you see a tall skinny mid-40s guy with blue Head bag, come over and say Hi... it could be yours truly :-)

What you are saying about tournament play at 3.5 level is very much true. But the guys who consistently make it deep into the draws and end up in a top-10 spot at the end of the year are good as 3.5 goes...

As for "you definitely cannot just show up at 4.0 and win 50% of your matches": I tried to stay away from personal anecdotes but here is one: several of my teammates were swept into the 4.0 ranks in the big bump-up of 2009. They were all good 3.5s but not that good... only had slightly over .500 records playing league at-level that year. It's hypothetical but if they all had played a lot of tourneys that year (which they didn't) it's unlikely that any of them would have made it into the top 10 in the state. Here we are a year later. They all have .500+ record in 4.0 leagues. Their team made it to the districts where all these guys had very decent showing. On an even more personal note, my dubs partner and I (both 3.5) played several friendly doubles matches against two of these guys lately. It was always very close (and fun) but we won all of these matches... So maybe you can just show up at 4.0 as a strong 3.5 and win half of your matches.

ian2
12-03-2010, 07:58 AM
Well, I have not played USTA in a couple of years, but still got bumped up last year. I was checking to see If I could have gotten bumped up again! At least this year, my rating did not change.

Interesting... so you were bumped up at the end of 2009 even though you didn't play one USTA match that year? I see how that happened: you were on the bubble and once they added .10 (or whatever that wholesale bump-up figure was that year), you made it over. Once again, it's explainable but ridiculous... yet another example of the lazy programming by NTRP system folks.

amarone
12-03-2010, 08:11 AM
Interesting... so you were bumped up at the end of 2009 even though you didn't play one USTA match that year? I see how that happened: you were on the bubble and once they added .10 (or whatever that wholesale bump-up figure was that year), you made it over. Once again, it's explainable but ridiculous... yet another example of the lazy programming by NTRP system folks.
Just why is it ridiculous? The USTA wanted to shift the boundaries between levels so people just below the boundary got bumped up. Why should someone escape that just because he has not played USTA for a year? His ability level is unlikely to have changed much.

ian2
12-03-2010, 08:11 AM
Sure I can email it to you when I get home from work. Keep in mind though this is before they changed the algorithm last year so I dont know how much of it is still valid but the document outlines how it was done pre eoy 2009.

:oops: I didn't give you my email... Here goes: ian2ten at gmail dot com. Thanks in advance!

ian2
12-03-2010, 08:18 AM
Just why is it ridiculous? The USTA wanted to shift the boundaries between levels so people just below the boundary got bumped up. Why should someone escape that just because he has not played USTA for a year? His ability level is unlikely to have changed much.

That's one way to look at it, and certainly a valid way. "Ridiculous" might have been too strong a word in this case. Another way to see this is that you "earn" your bump-up. If you were say a 4.0 in 2008 and did not play a match in 2009 you are still a 4.0, no matter what relative movement was applied to other people who did play... OK this is not one of the patently absurd examples, it's merely a questionable one.

813wilson
12-03-2010, 08:24 AM
3 examples:

A doubles team has gone undefeated the last two seasons. 15 total matches/3 of which went three sets. Two wins this year against players who were bumped up and they stayed put.

One guy went 7-1 two seasons ago. all straight set wins. loss was a 3rd set tiebreaker and finished the match on a sprained ankle. He wgot bumped up. Sat out last season and was moved back down.

know a guy who played doubles exclusively. went 5 and 1 with the same partner. Partner was bumped and he stayed put.

OrangePower
12-03-2010, 08:37 AM
As for "you definitely cannot just show up at 4.0 and win 50% of your matches": I tried to stay away from personal anecdotes but here is one: several of my teammates were swept into the 4.0 ranks in the big bump-up of 2009. They were all good 3.5s but not that good... only had slightly over .500 records playing league at-level that year. It's hypothetical but if they all had played a lot of tourneys that year (which they didn't) it's unlikely that any of them would have made it into the top 10 in the state. Here we are a year later. They all have .500+ record in 4.0 leagues. Their team made it to the districts where all these guys had very decent showing. On an even more personal note, my dubs partner and I (both 3.5) played several friendly doubles matches against two of these guys lately. It was always very close (and fun) but we won all of these matches... So maybe you can just show up at 4.0 as a strong 3.5 and win half of your matches.

I agree with the bolded part as conventional wisdom. What your anecdote demonstrates as that in these post-great-bump ages, conventional wisdom is out of the window for a few years. Your ex-3.5 teammates were successful at 4.0 only because 4.0 is the new 3.5... most of the mid to top 4.0s were bumped up to 4.5 as your friends were moved to 4.0.

Thunderbrat
12-03-2010, 08:45 AM
Some things to keep in mind when you think about ratings:

Win/Loss records mean nothing. It's all about the games. For example if you lose a match 0-6, 7-6, 1-0 (TB) the computer sees it as 12-8 in your favor and will adjust ratings accordingly!

A few singles matches can have a major inpact on a doubles player. This is because doubles match results are diluted by the fact that the DNTRPs of all four players are invloved. Also note that people tend to hold serve in doubles even when they are overmatched, so while 6-3, 6-2 is a real thrashing in doubles, it's a pretty standard singles score and these commonly lopsided singles scores can move people fast.

Your results against "B" benchmarked players (those that go to nationals/sectionals) count heavier than results against other players. Even your results against players that played against players that are benchmarked have added weight. Do well against these players and your chances of getting bumped go way up!

The USTA has stated that they've adjusted the computer the last few years in an attempt to equal out the number of players at various levels. Translation? There are way to many 3.5 and 4.0 players. Lets adjust the threshhold so that many 3.5's move to the 4.0 level and some of the the top 4.0s become 4.5s.

This move by the USTA has been controversial but I think it's is a good idea. I was just bumped to 5.0 and we are really struggling to form a league because there just aren't enough players at that level willing to play. In the past players that got bumped to 5.0 would either have to give up on league play or go sandbag in a few tournaments to try to get back to 4.5. If a few more of the top 4.5s join us we could have a nice league of our own!

ian2
12-03-2010, 08:56 AM
I agree with the bolded part as conventional wisdom. What your anecdote demonstrates as that in these post-great-bump ages, conventional wisdom is out of the window for a few years. Your ex-3.5 teammates were successful at 4.0 only because 4.0 is the new 3.5... most of the mid to top 4.0s were bumped up to 4.5 as your friends were moved to 4.0.
Yes but this is the age we are in. We are left to deal with the reality of it...

ian2
12-03-2010, 09:29 AM
Some things to keep in mind when you think about ratings:

Win/Loss records mean nothing. It's all about the games. For example if you lose a match 0-6, 7-6, 1-0 (TB) the computer sees it as 12-8 in your favor and will adjust ratings accordingly!

A few singles matches can have a major inpact on a doubles player. This is because doubles match results are diluted by the fact that the DNTRPs of all four players are invloved. Also note that people tend to hold serve in doubles even when they are overmatched, so while 6-3, 6-2 is a real thrashing in doubles, it's a pretty standard singles score and these commonly lopsided singles scores can move people fast.

Your results against "B" benchmarked players (those that go to nationals/sectionals) count heavier than results against other players. Even your results against players that played against players that are benchmarked have added weight. Do well against these players and your chances of getting bumped go way up!

The USTA has stated that they've adjusted the computer the last few years in an attempt to equal out the number of players at various levels. Translation? There are way to many 3.5 and 4.0 players. Lets adjust the threshhold so that many 3.5's move to the 4.0 level and some of the the top 4.0s become 4.5s.

This move by the USTA has been controversial but I think it's is a good idea. I was just bumped to 5.0 and we are really struggling to form a league because there just aren't enough players at that level willing to play. In the past players that got bumped to 5.0 would either have to give up on league play or go sandbag in a few tournaments to try to get back to 4.5. If a few more of the top 4.5s join us we could have a nice league of our own!

Thunderbrat, you are explaining how the system works currently. I know how the system works (to the extent possible, given the secrecy of NTRP algorithm); what I'm arguing is that the way it works is broken, in some respects.

"Win/Loss records mean nothing. It's all about the games". Yes, but should it be that way? Win/loss should be a factor in the algorithm in my opinion. Otherwise you see the numerous examples where a player who did not win a single match at level might be bumped up and someone with 0.900+ record is not. Then the player who got bumped up continues his losing ways at the higher level, with nary a chance to be bumped down as long as he/she wins a few games here and there. While the player who stayed can kill it again next year and still not get bumped up despite dominating the competition at level.

"A few singles matches can have a major impact on a doubles player." Again, true. But the way I see it, this is an argument for separate singles and doubles ratings. Very nice observation BTW about how the differences in singles vs dubs style of play contribute to this inequality.

"Your results against "B" benchmarked players (those that go to nationals/sectionals) count heavier": true again. Problem is (again, just my opinion, and difficult to substantiate since the details of NTRP algorithm are kept secret) - the actual "weighting" is off. I.e. the weight given to matches against these players appears to be disproportionally high. The rationale behind the "ripping effect" you refer to is also questionable.

As for USTA intentions at re-balancing the levels: I actually agree with what they have done. This has little to do with the system deficiencies however: they just applied "targeted post-production adjustments" to the results generated by the deficient system...

Spokewench
12-03-2010, 10:51 AM
Here's one I don't understand.

Player is a 3.5 rated player - due to living in rural area, does not play that much.

Player played as a 3.5 player in 4.0 adult league for 3 matches.

1st Match Played with 3.5 player (who got bumped this year to 4.0) against a 3.5 and a middle of road 4.0, lost 2 - 2

2nd Match played with middle of road 4.0 player against middle of road 4.0 and higher level 4.0 lost 2, 3.

3rd Match played with 4.0 player against two upper level 4.0s (both of whom moved to 4.5 this year) lost 2, 3.

How does this player move from 3.5 to 4.0?

OrangePower
12-03-2010, 11:09 AM
^^^^^

Something to add that might apply to this case above but is also something to generally keep in mind...

Your likelyhood of getting bumped up/down also depends on your exact rating at the start of the year. For example, if you started the year at 3.49 (so your published rating would have been 3.5), even a small increase would push you into 4.0 territory (3.51+). But had you started at 3.01 (also a published 3.5), played the same opponents, and obtained the same results, there might not have been enough of an increase to get bumped.

So in the case above, maybe this guy was already on the cusp anyway, so not much needed to get the bump.

amarone
12-03-2010, 11:13 AM
Here's one I don't understand.

Player is a 3.5 rated player - due to living in rural area, does not play that much.

Player played as a 3.5 player in 4.0 adult league for 3 matches.

1st Match Played with 3.5 player (who got bumped this year to 4.0) against a 3.5 and a middle of road 4.0, lost 2 - 2

2nd Match played with middle of road 4.0 player against middle of road 4.0 and higher level 4.0 lost 2, 3.

3rd Match played with 4.0 player against two upper level 4.0s (both of whom moved to 4.5 this year) lost 2, 3.

How does this player move from 3.5 to 4.0?
Sounds absolutely right to me. In match 2 he and his partner took 5 games off mid/high 4.0s. In match 3 they took 5 games off players who were at or close to 4.5. Remember, a bottom of range player is expected to lose 0 & 0 to a player at the very top of the same range. Doubles complicates things, but at least two of those three results suggest that the guy should indeed be a 4.0.

amarone
12-03-2010, 11:16 AM
^^^^^

Something to add that might apply to this case above but is also something to generally keep in mind...

Your likelyhood of getting bumped up/down also depends on your exact rating at the start of the year. For example, if you started the year at 3.49 (so your published rating would have been 3.5), even a small increase would push you into 4.0 territory (3.51+). But had you started at 3.01 (also a published 3.5), played the same opponents, and obtained the same results, there might not have been enough of an increase to get bumped.

So in the case above, maybe this guy was already on the cusp anyway, so not much needed to get the bump.
Exactly. And also bear in mind that the ratings are calculated based on the exact ratings of all players involved at the time. Hence 4.0 players "who were later bumped to 4.5" may well have had a 4.5 rating at the time of the match.

And also bear in mind that everyone in a local/district/section can be moved slightly as a result of the benchmark process when the EOY ratings are produced.

Spokewench
12-03-2010, 11:34 AM
So, can you explain the Benchmark process to me? I got moved to 3.5 this year and am now a Benchmark. My friend got moved to 4.0 this year and is also a Benchmark. we both played in sectionals at 3.5 level.

What makes a person a benchmark?

cknobman
12-03-2010, 11:39 AM
So, can you explain the Benchmark process to me? I got moved to 3.5 this year and am now a Benchmark. My friend got moved to 4.0 this year and is also a Benchmark. we both played in sectionals at 3.5 level.

What makes a person a benchmark?

Anyone who plays in playoffs at any level (city, sectional, state, national) will become a benchmark player.

Since you both made it to sectionals this is why you are both now "Benchmark" players.

amarone
12-03-2010, 11:47 AM
So, can you explain the Benchmark process to me? I got moved to 3.5 this year and am now a Benchmark. My friend got moved to 4.0 this year and is also a Benchmark. we both played in sectionals at 3.5 level.

What makes a person a benchmark?
Playing in playoffs at some levels makes you a benchmark. It is called that (I assume) because these players are used as a "benchmark" for their section/district/local league.

The annual process starts by looking at the results at nationals. Here, players from different sections have played each other and so can be used to calibrate the levels for those sections across the country. As an example, let us say that the Southern 4.0 champions, with an average exact rating of 3.92, beat up on NorCal, who had an average rating of 3.95. That suggests that the Southern ratings are low in comparison with NorCal. The end-of-year process takes this into account and there will be some adjustment of 4.0 players to compensate for this. Obviously it is more complicated than that simple example as there are many sections playing each other.

Then they look at the section results. Southern is made up of nine districts, so the process looks at the results at sectionals compared to the ratings and again recalibrates.

Wash/rinse/repeat at district and local levels, and that is how the benchmarks are used to help calibrate ratings across all levels nationally.

My wife just got dropped from 4.5 to 4.0, despite having a good record at 4.5. She knows other players to whom this has happened. I suspect, therefore, that the benchmark process resulted in a lowering of 4.5 women in Atlanta.

dizzlmcwizzl
12-03-2010, 04:09 PM
There is a guy that plays for another team in our district. He is a very good player in practice but always manages to choke when the points count for something. He often looses close matches to whomever he plays (very good or not).

Two seasons ago he went winless on the season. The coach said it was because he arrived severely hung over for every Saturday AM match. This season he did manage to win two 4.0 matches going 2-2 on the season in 4. But his team did not play him in local playoffs. Additionally, he hooked up with a 4.5 team and on that team he went 0-3 against 4.5 players ... However he and his 4.5 partner but did manage a win against a team of 4.0 players.

So over the last two years 3-7 at 4.0 and 1-4 at 4.5 .... So what happens .... he gets bumped up!

Beagle97
12-03-2010, 04:16 PM
Maybe that explains why I got bumped. I played 5 mixed doubles matches as a 3.5 self-rating, and thought only 2 would count, because the rest were against other self-rated players. But one of the matches, the other self-rated person ended the year going to districts in the 3.5 adult league, and got bumped to 4.0b.

So if they count my match against that one team as against a benchmark player, it would show that I was competitive with a 4.0b player? And then bump me to 4.0m?

The only other thing I can make out, is that one of the 2 matches that had computer rated opponents, one of the opponents was bumped to 4.0C from 3.5C.

I think they are more strict against self-rated players. Probably with good reason. However when a person will get bumped is still a mystery to me.

damazing
12-03-2010, 06:22 PM
I found an example of a former teammate of mine that didn't get bumped but many of the people he beat got bumped - here's his record:

Singles
Won 6-3, 7-6 (opponent got bumped up)
Won 4-6, 6-0, 1-0
Won 2-6, 6-2, 1-0 (opponent got bumped up)

Doubles
Lost 7-5, 6-2
Lost 7-6, 6-3 (both opponents got bumped up)
Won 6-4, 6-2 (My team mates partner this match was .5 level lower)
Lost 6-4, 6-3 (one of the opponents got bumped up)((my team mates partner this match was .5 level lower)

Based on three doubles losses he didn't get bumped up even though two of the losses were to players that got bumped up and the scores were still competitive.

I would have thought he would have gotten extra dntrp points for being competitive with a partner that was .5 level lower and the close scores.

He's clearly at the top end of the level with his singles results showing he could compete - should he just not play any doubles next year if he wants to get bumped?

Jim A
12-03-2010, 06:58 PM
- yes, absolutely, in the sense that these top 3.5s will beat bottom 25% of 4.0s 90% of the time and will beat lower 50% of 4.0s half of the time. Sorry about all the fractions :-)

Ian, I used the 50% based on your quote above...again we're not naming names but I can easily look through tennislink see people who were bumped this week and played up in Twilight only to see them have horrible records for the fall....

At least 3 of the players you were discussing have been bumped to 4.0 when I look them up individually

anyway...a little more digging around tennislink and I found some league results at level for some of them as well...

2-4, 0-2, 2-3,

NoSkillzAndy
12-03-2010, 08:09 PM
People choose to play up, for a variety of reasons. There is nothing wrong with that. The wrong thing is that the algorithm cannot handle these situations. It demonstrably breaks down under such conditions and produces results that range from borderline "reasonable" to patently absurd.

What is wrong with the algorithm? If the player plays up and is competitive, then that person should play in that level. It is pretty simple. How does it "demonstrably breaks down"?

Allow me to post the example I gave in the other thread of a 3.5 player playing 4.5 league for the past few years and getting bumped up to 4.0 at year-end two years ago and up to 4.5 this year. Hopefully the algorithm was NOT intended to work this way...

Complete 2010 Results:
1-6, 0-6 #2 Singles 4.5
0-6, 2-6 #3 Doubles 4.5
0-6, 1-6 #3 Doubles 4.5

He hasn't won a USTA match since 2001 (unless you count a forfeit). However, since he had such a stellar year last year, I'm sure all he had to do this year was win 4 games in 3 matches to secure his bump-up to 4.5 :rolleyes:

Complete 2009 Results:
0-6, 1-6 #2 Mixed Combo 8.0
1-6, 0-6 #3 Mixed Combo 8.0
6-7, 1-6 #2 Mixed Combo 8.0
1-6, 3-6 #2 Mixed Combo 8.0
1-6, 3-6 #2 Mixed Combo 8.0
2-6, 2-6 #2 Mixed Combo 8.0
3-6, 2-6 #3 Doubles 4.5
1-6, 2-6 #3 Doubles 4.5
6-4, 2-6, 0-1 #3 Doubles 4.5
1-6, 0-6 #3 Doubles 4.5
0-6, 1-6 #3 Doubles 4.5
0-6, 2-6 #3 Doubles 4.5
6-0, 6-0 default #3 Doubles 4.5
4-6, 0-6 #3 Doubles 4.5
2-6, 2-6 #3 Doubles 4.5
0-6, 2-6 #1 Doubles 4.5
3-6, 2-6 Tournament Singles 4.0

NoSkillzAndy
12-03-2010, 08:21 PM
Sure I can email it to you when I get home from work. Keep in mind though this is before they changed the algorithm last year so I dont know how much of it is still valid but the document outlines how it was done pre eoy 2009.

I would love to get a copy of that as well. If it's not too much trouble can you email it to me too?

amarone
12-04-2010, 03:02 AM
Maybe that explains why I got bumped. I played 5 mixed doubles matches as a 3.5 self-rating, and thought only 2 would count, because the rest were against other self-rated players. But one of the matches, the other self-rated person ended the year going to districts in the 3.5 adult league, and got bumped to 4.0b.

So if they count my match against that one team as against a benchmark player, it would show that I was competitive with a 4.0b player? And then bump me to 4.0m?

The only other thing I can make out, is that one of the 2 matches that had computer rated opponents, one of the opponents was bumped to 4.0C from 3.5C.

I think they are more strict against self-rated players. Probably with good reason. However when a person will get bumped is still a mystery to me.
The fact you played a benchmark player is of no relevance. Benchmark just means that s/he played in playoffs.

Matches against self-rated players do count as long as the self-rated player has played at least two matches that count for ratings. At this point, the player will have a dynamic rating that will be used in any match played to calculate the new dynamic ratings of him/her and opponents.

amarone
12-04-2010, 03:08 AM
I found an example of a former teammate of mine that didn't get bumped but many of the people he beat got bumped - here's his record:

Singles
Won 6-3, 7-6 (opponent got bumped up)
Won 4-6, 6-0, 1-0
Won 2-6, 6-2, 1-0 (opponent got bumped up)

Doubles
Lost 7-5, 6-2
Lost 7-6, 6-3 (both opponents got bumped up)
Won 6-4, 6-2 (My team mates partner this match was .5 level lower)
Lost 6-4, 6-3 (one of the opponents got bumped up)((my team mates partner this match was .5 level lower)

Based on three doubles losses he didn't get bumped up even though two of the losses were to players that got bumped up and the scores were still competitive.

I would have thought he would have gotten extra dntrp points for being competitive with a partner that was .5 level lower and the close scores.
That is what the algorithm does, but note two things:


The player's dynamic rating may not be 0.5 lower; it could be 0.1 lower and just cross the boundary into the next half level down.
At the time the match was played, the lower player could even have a dynamic ranking that is much higher based on his/her record since the last ratings run.


He's clearly at the top end of the level with his singles results showing he could compete - should he just not play any doubles next year if he wants to get bumped?Singles players do tend to get bumped more easily as there is no partner rating to confuse things.

OrangePower
12-04-2010, 08:02 AM
I found an example of a former teammate of mine that didn't get bumped but many of the people he beat got bumped - here's his record:

Singles
Won 6-3, 7-6 (opponent got bumped up)
Won 4-6, 6-0, 1-0
Won 2-6, 6-2, 1-0 (opponent got bumped up)

Doubles
Lost 7-5, 6-2
Lost 7-6, 6-3 (both opponents got bumped up)
Won 6-4, 6-2 (My team mates partner this match was .5 level lower)
Lost 6-4, 6-3 (one of the opponents got bumped up)((my team mates partner this match was .5 level lower)

Based on three doubles losses he didn't get bumped up even though two of the losses were to players that got bumped up and the scores were still competitive.

I would have thought he would have gotten extra dntrp points for being competitive with a partner that was .5 level lower and the close scores.

He's clearly at the top end of the level with his singles results showing he could compete - should he just not play any doubles next year if he wants to get bumped?

That is what the algorithm does, but note two things:


The player's dynamic rating may not be 0.5 lower; it could be 0.1 lower and just cross the boundary into the next half level down.
At the time the match was played, the lower player could even have a dynamic ranking that is much higher based on his/her record since the last ratings run.


Singles players do tend to get bumped more easily as there is no partner rating to confuse things.

What amarone said, plus:
3. It also depends on the dynamic rating going into the year; meaning, those guys that got bumped may have started the year close to the cutoff for the next level, so needed only a relative small increase to get bumped, whereas the guy that didn't get bumped may have started at the very bottom of the level, so would have needed more of an overall increase to cross the boundary to the next level.

MrCLEAN
12-05-2010, 06:16 PM
A player was 4-13 in 3.5S, got moved to 4.0 at the end of last year. Was 0-11 this year in 4.0S. 2010 year end ranking....4.0

40Love
12-05-2010, 09:04 PM
3.5 self rate playing 4.0 adult in KY. Played two matches (League career). Won first 6-4, 7-5. Won match two 6-1, 6-3 against another 3.5 playing up. Year end rating = 4.5

dlk
12-05-2010, 09:06 PM
3.5 self rate playing 4.0 adult in KY. Played two matches (League career). Won first 6-4, 7-5. Won match two 6-1, 6-3 against another 3.5 playing up. Year end rating = 4.5

Wow, now that's a bump. Would like to see his record this year.

ian2
12-06-2010, 06:44 AM
I've seen some bizarre stuff personally but ratings examples posted by NoSkillzAndy and 40Love take the cake, by far...

Here is one more that I've come across: 3.0C player at the start of 2010.
3.0 stats for the year: 6-3 singles, 3-1 dubs.
3.5 stats for the year: 3-2 singles, 8-3 dubs.
Did not play any matches above 3.5 level.
Year-end rating: 4.0

gameboy
12-06-2010, 07:26 AM
This is what is really wrong with USTA league play today - not the computer ranking, but people who care too much about W-L record (which lead to sandbagging, cheating, etc.).

This is a recreational league, not some US Open Qualifier. USTA's aim is create a fun atmosphere where people of similar skills are matched against another. Which is exactly what the computer rating system does. It doesn't care about your W-L record that much. What it cares about is how competitive you are against your opponent.

All the bellyaching done on this thread is PURELY based on W-L record and nothing else. But why complain about how computer is not using the W-L record when it was never designed for that purpose in the first place?

Stop caring about your record so much! USTA levels are purely artificial constructs and arbitrary. Just by luck your skills can land you at a top of a level or at the bottom of the next level.

Just go out and have fun. And if you cannot have fun unless you win all the time, I suggest you go out and find some other sport.

dlk
12-06-2010, 07:33 AM
This is what is really wrong with USTA league play today - not the computer ranking, but people who care too much about W-L record (which lead to sandbagging, cheating, etc.).

This is a recreational league, not some US Open Qualifier. USTA's aim is create a fun atmosphere where people of similar skills are matched against another. Which is exactly what the computer rating system does. It doesn't care about your W-L record that much. What it cares about is how competitive you are against your opponent.

All the bellyaching done on this thread is PURELY based on W-L record and nothing else. But why complain about how computer is not using the W-L record when it was never designed for that purpose in the first place?

Stop caring about your record so much! USTA levels are purely artificial constructs and arbitrary. Just by luck your skills can land you at a top of a level or at the bottom of the next level.

Just go out and have fun. And if you cannot have fun unless you win all the time, I suggest you go out and find some other sport.

You, sir, are one of the most cynical posters on this site. You seem to always condemn everyone, while talking down to us like you're our all-knowing father:confused:

gameboy
12-06-2010, 07:41 AM
You, sir, are one of the most cynical posters on this site. You seem to always condemn everyone, while talking down to us like you're our all-knowing father:confused:

I speak out against people complaining about cheating, ranking, and those who generally take "recreational" leagues too seriously.

There are just too many people like that posting around here.

dlk
12-06-2010, 07:43 AM
I speak out against people complaining about cheating, ranking, and those who generally take "recreational" leagues too seriously.There are just too many people like that posting around here.

In the same vein, don't take the posts so seriously.:)

ian2
12-06-2010, 08:52 AM
I speak out against people complaining about cheating, ranking, and those who generally take "recreational" leagues too seriously.

There are just too many people like that posting around here.

Since I started this thread I feel obliged to respond. This is not about cheating or having fun or sandbagging or recreational vs pro, etc. This is about DNTRP algorithm and the math and statistics involved. I hope you agree that some of the examples provided in this thread and a couple of similar threads illustrate beyond any doubt that the system is way out of whack in some respects?

And just to clear the air: my premise is the opposite of what you are implying. My concern is that under certain common scenarios the system bumps up people who did not deserve to be bumped (this diluting the higher level of play) while penalizing (holding back) people who do. The main contributing factor here is that results of playing up are weighted disproportionately higher than results of playing at-level. Another contributing factor is that, as you said, "it's not about Win/Loss". I agree that it should not be all about W/L or even mostly about W/L. But the fact that W/L record is completely disregarded by the existing system results in some absurd outcomes. Would you agree that example of someone who never won a match in years moving up the ranking ladder while someone winning 90% of they matches staying put represents just that: and absurd outcome?

BTW, the much ballyhooed "mass bump-up of 2009" was for the most part an attempt to correct the phenomenon of players "accumulating" at the higher ranges of the levels with little chance of ever crossing over, due to the math involved - which was an outcome built into the system! Problem is, all the USTA did was a one-time correction of the outcomes. They did nothing to fix the system producing such outcomes.

tennisjon
12-06-2010, 09:12 AM
Three years ago my friend got bumped down from 4.5 to 4.0 even though he went 2-0 in doubles with me that season in his only matches. Meanwhile 2 years ago, a guy in my league went winless in 4.5 and 4.0 and got bumped up to 4.5. This year a guy in my league went undefeated (in his first season being rated) and easily won his matches against good competition, yet didn't get bumped up, while a guy on my team got bumped to 5.0 even though he won only about 2/3 of his matches.

rnitz
12-06-2010, 01:36 PM
I don't want to defend the NTRP rating examples because some of them really do seem to be ridiculous, but it may help to remember that the computer is not just looking at the match score, but the score taking into account the opponents strength.

Elisa and cknobman said it earlier, but if your playing a better player player and you lose, you still may have your rating improve if you lost by less than what the computer thought you should lose by.

A few years ago one of the sections went into detail about the NTRP algorithm at the time, and if I remember correctly, the computer expected you to lose by one game for every .03 difference in player ratings (this was before the recent NTRP changes, so it may not apply to what they're doing now). For example, if a 4.20 player played a 4.32 opponent the computer expected you to lose 4-6, 4-6 (4 games x .03 = 0.12 difference in player ratings). So if you're a 4.10 player and you play a 4.45 player and you lose 1-6, 2-6 your rating will actually go up (because the computer expected you to lose 0-6, 1-6 given the 0.35 rating differential of the players).

As Ian2 noted, this becomes absurd when players play up a level and have very large differentials with opponents. Imagine a 4.0 player (3.90 rating) playing up against the 4.45 player and getting killed - he would still go up in ratings (and maybe get bumped). I think NTRP needs to have a better method of handling players that play up a level.

I'll see if I can find a link to the detailed description of the NTRP algorithm (from 5 or 6 years ago) if anyone's interested.

rnitz
12-06-2010, 01:42 PM
Found the link to the NTRP rating algorithm discussion on a web archive (hope the link comes through):

http://web.archive.org/web/20051103100413/www.wetennis.com/rate.htm

ian2
12-06-2010, 01:57 PM
Found the link to the NTRP rating algorithm discussion on a web archive (hope the link comes through):

http://web.archive.org/web/20051103100413/www.wetennis.com/rate.htm

Thanks! The link works and this is a very useful info, with a few caveats:

1. It is not an "official" document; the author described his understanding of the algorithm.

2. The info is out of date (last updated in 2005) but the current system likely works very much the same way.

Anyway, great find, thanks again for providing the link!

SweetH2O
12-06-2010, 04:57 PM
As Ian2 noted, this becomes absurd when players play up a level and have very large differentials with opponents. Imagine a 4.0 player (3.90 rating) playing up against the 4.45 player and getting killed - he would still go up in ratings (and maybe get bumped). I think NTRP needs to have a better method of handling players that play up a level.

It's clear that NTRP is not a perfect system, so I'm not defending it. But I think it's also up to the players to self-police themselves a bit too.

Players shouldn't be playing up unless they feel they can be competitive at the higher level and give their opponents a good match. And if that's the case, they should be expecting to be bumped up. Playing up should be the stepping stone to getting bumped up.

I think it has become too much the norm that players think it's their right to play at level and up a level without consequence. This is where more self-policing needs to be involved instead of relying on a computer to do it for them.

athiker
12-07-2010, 07:49 AM
I don't want to defend the NTRP rating examples because some of them really do seem to be ridiculous, but it may help to remember that the computer is not just looking at the match score, but the score taking into account the opponents strength.

....

As Ian2 noted, this becomes absurd when players play up a level and have very large differentials with opponents. Imagine a 4.0 player (3.90 rating) playing up against the 4.45 player and getting killed - he would still go up in ratings (and maybe get bumped). I think NTRP needs to have a better method of handling players that play up a level.

I'll see if I can find a link to the detailed description of the NTRP algorithm (from 5 or 6 years ago) if anyone's interested.

It's clear that NTRP is not a perfect system, so I'm not defending it. But I think it's also up to the players to self-police themselves a bit too.

Players shouldn't be playing up unless they feel they can be competitive at the higher level and give their opponents a good match. And if that's the case, they should be expecting to be bumped up. Playing up should be the stepping stone to getting bumped up.

I think it has become too much the norm that players think it's their right to play at level and up a level without consequence. This is where more self-policing needs to be involved instead of relying on a computer to do it for them.

I tend to agree with the second quote and think the algorithm should err on the side of bumping people up that play up. Why would they be playing up unless they feel they can at least have a decent match and not be completely embarrassed and a waste of time for their opponents? What's the cost for these players to get bumped up? They will simply be playing where they are already playing anyway...they just can't "play down" anymore at their old rating...double dip so-to-speak.

I had a match this year where both of the opponents were "playing up". We destroyed them and it was pretty much a waste of an evening. I had more fun losing other matches. I don't have a problem with players playing up but if a person is going to play up they should feel like they are on the cusp of being bumped up anyway. If they get bumped then no big surprise.

ian2
12-07-2010, 10:37 AM
I tend to agree with the second quote and think the algorithm should err on the side of bumping people up that play up. Why would they be playing up unless they feel they can at least have a decent match and not be completely embarrassed and a waste of time for their opponents? What's the cost for these players to get bumped up? They will simply be playing where they are already playing anyway...they just can't "play down" anymore at their old rating...double dip so-to-speak.

I had a match this year where both of the opponents were "playing up". We destroyed them and it was pretty much a waste of an evening. I had more fun losing other matches. I don't have a problem with players playing up but if a person is going to play up they should feel like they are on the cusp of being bumped up anyway. If they get bumped then no big surprise.

When I started this thread my intention was to compile a "catalog" of examples where NTRP calculations resulted in nonsensical outcomes for a given "input data". I'd like to thank everyone who contributed. I'd appreciate any additional examples but I hope most would agree that what's been posted so far shows beyond any doubt that the system is broken in some respects. A few of the examples were quite gaudy... far worse than anything I've happen to run across personally or while snooping around at random on Tennislink.

I also asked for examples only and no discussion in this particular thread. That proved to be an unreasonable request on my part. Apparently people are not self-policing, starting with myself :-) Many posters wanted to express their opinions and I appreciate everyone's contribution.

athiker, in regard to some specific points in your post:

"I ... think the algorithm should err on the side of bumping people up that play up". I'm continuously surprised that many players apparently see bump-up as a punishment. I just don't get this. I see it as a validation that all the work one have put in to improve their game have paid off. But at the same time the bump-up should be earned. Currently the algorithm indeed errs on the side of bumping people up that play up. And in some cases, as shown in this thread, it errs very badly, by bumping up people who did not in any way deserve to be bumped up. It also errs in the other direction, by not bumping up people who did "deserve" a bump-up. Ideally the algorithm should not err on either side. Well I know, "ideally." But is it too much to ask that at least it did not err SPECTACULARLY? You hear me USTA?

"Why would they be playing up unless they feel they can at least have a decent match and not be completely embarrassed... if a person is going to play up they should feel like they are on the cusp of being bumped up anyway". I don't know about this "feelings" and "embarrassment" stuff... As a personal anecdote, early this year I was considering playing up (way up, 3.5 playing in 4.5 and 5.0 draws). I felt uncomfortable about "wasting time" of better players, and asked opinions of 4.5+ players in this forum. Almost everyone was supportive. The "worst" I heard was "I don't really care but don't mind having an easy first-round match". I ended up playing one tournament match against a strong 4.5 player and two matches against 5.0 players this year. The results were predictable (0 and 1, 1 and 0, 0 and 0) but each of these guys was extremely supportive about my foray into the higher levels of play... or at least polite :-) What I'm getting to: people play up for a number of reasons, some "good" and some, well, I'm not sure. I know of a 3.0 guy who entered an Open tourney and now calls himself "an Open level player"... it cost him $75 so I guess he earned the honors :-) But people do play up, whatever the motivation. The system should be able handle such situations in a reasonable manner. Currently it doesn't: it puts too much weight on someone winning a few games here and there while playing up.

athiker
12-07-2010, 06:57 PM
Sorry, I was just responding to the post in front of mine...missed or forgot the earlier no commentary request. :) My only frame of reference is USTA league play, singles Compass Draws and Ladders. I've never played a tournament and think that's a bit different b/c I think that's more of an all day or two day event where one expects to play several rounds. As you say, if an early round is easy in that scenario, not big deal.

It wasn't that big of a deal in the doubles league either, just kind of a wasted night, far from the end of the world. League play is only once per week though and with the number of players on our team we only played on average every other week. Guess that's why people like to play up...so they can play on more than one team and play every week! I do know I avoided one of the players in our recent ladder b/c I knew he was "playing up" and wasn't competitive with anyone else. I don't have a lot of free time with family obligations and such so like to maximize the fun of my competitive outlets.

I agree that I'm surprised when people are disappointed at getting bumped up...my goal is to get bumped up. Though I am a bit mystified why one guy in our group got bumped up, as is he, but he was already playing up exclusively so it didn't really matter. So there is the main reason folks probably play up...just to hang with their tennis buddies. I guess another is to challenge themselves if they feel they are improving and figure that will help speed things along...heck maybe that will be me this spring!

ian2
12-08-2010, 11:10 AM
athiker, no need to apologize. As I said, my request for examples only and no discussion was unreasonable in the first place... this is a discussion board after all :-) Thank you for your contribution!

Beagle97
12-14-2010, 02:18 PM
Wow, I just found out one of my teammates from last year got bumped from 3.5s to 4.0m, having played 2 matches for the year. He's definitely not a 4.0 player.

That's his entire history with USTA, 2 7.0 mixed doubles matches in 2010.

First match he lost, 2-6, 6-4, 0-1

Second match he lost, 2-6, 4-6

He played with 2 partners, both 3.5C who remain 3.5C in 2011.

Of the opponents, the first match guy (3.5C) remains 3.5C for 2011. The girl (3.0T) went to 4.0M for 2011, after playing 7 matches in 7.0 mixed doubles for 2010, with a record of 5-2.

The second match the guy and girl, both 3.5C remain 3.5C for 2011.

How they put my team mate up to 4.0 based on so little is a little unbelievable to me. The funny thing (not so funny) is that he spoke with our captain beforehand, being afraid of being bumped to 4.0--he didn't want to play all out. We were like...play all out! You are barely competitive in 7.0 mixed doubles!

ian2
12-15-2010, 07:43 AM
Wow, I just found out one of my teammates from last year got bumped from 3.5s to 4.0m, having played 2 matches for the year. He's definitely not a 4.0 player.

That's his entire history with USTA, 2 7.0 mixed doubles matches in 2010.

First match he lost, 2-6, 6-4, 0-1

Second match he lost, 2-6, 4-6

He played with 2 partners, both 3.5C who remain 3.5C in 2011.

Of the opponents, the first match guy (3.5C) remains 3.5C for 2011. The girl (3.0T) went to 4.0M for 2011, after playing 7 matches in 7.0 mixed doubles for 2010, with a record of 5-2.

The second match the guy and girl, both 3.5C remain 3.5C for 2011.

How they put my team mate up to 4.0 based on so little is a little unbelievable to me. The funny thing (not so funny) is that he spoke with our captain beforehand, being afraid of being bumped to 4.0--he didn't want to play all out. We were like...play all out! You are barely competitive in 7.0 mixed doubles!

Thanks Beagle, this is a good one! As anyone who read through this thread knows, my view is that NTRP algorithm is broken. This example is yet another illustration of just how badly it is broken. So, how did the system "decide" to bump up this player? I mean there is no omnipotent deity sitting there deciding who gets to be moved up, who gets to be to moved down, and who gets to stay put. It's all math and statistics, right? Examples in this thread allow an insight into how this (insane) logic works. I too might be delusional but I think I'm beginning to see the method to this madness... not that it makes it any less absurd. Here are a few theories to explain the outcome of this specific example:

Theory #1: The algorithm puts all self-rated players at the upper end of the corresponding NTRP range, on the assumption that they are all potential sandbaggers. This player was self-rated at the start of the year, so he was assigned a "high 3.5" rating initially, let's say 3.49. Even a minor increase in DNTRP would push him over the bubble.

Theory #2: Self-rated players are assigned median NTRP initially (around 3.25 in this case) but during the end-of-year calculations all self-rated players get a certain "co-efficient" added to their rating, again on the assumption that they are potentially sandbagging. Combined with a minor "earned" increase in DNTRP it would push this player's DNTRP into 3.50+ territory.

There is another thing in play here: I believe it had been already established that the less "history" you have the larger your incremental rating change will be when it is calculated after a match. This player had no history at the start of the year (self-rated) and only played two matches which resulted in disproportionally large increase in his DNTRP. Had he played more, his DNTRP might have "leveled out" over time; with only two matches his DNTRP "spiked" helping to push him over the bubble.

These are just guesses but I think I'm on to something here. And even if I'm wrong in the particulars, it is absolutely clear that DNTRP system is very poorly "calibrated". As I said before, it tends to bump up players who did not "deserve" it while holding back players who did.

This example also shows a different aspect of the problems with the algorithm: the system draws statistical conclusions from a very small sample where no such conclusions could be made. In this case it had to do with an extremely small number of outcomes (two matches) but I also suspect a similar problem with "benchmark" adjustments: major adjustments are made based on a very small sample of "benchmark" players... and this sample might be the least representative of a given level of play in the first place!

kylebarendrick
12-15-2010, 01:45 PM
I'm pretty sure that self-rates aren't assigned an NTRP rating at all. The self-rating just lets them play at a certain level. Their dynamics are based purely on their match results on not on their self rating. I agree that ratings based on a small number of matches will vary wildly due to the poor statistics. I think that's just the nature of the beast since it is better to calculate some rating rather than none at all if possible.

"Benchmarks' may play into this last example also. If the 3.5 team from their section did well at nationals, then that would tend to drive up the ratings for all the 3.5s in that section.

ian2
12-15-2010, 01:58 PM
I'm pretty sure that self-rates aren't assigned an NTRP rating at all. The self-rating just lets them play at a certain level. Their dynamics are based purely on their match results on not on their self rating. I agree that ratings based on a small number of matches will vary wildly due to the poor statistics. I think that's just the nature of the beast since it is better to calculate some rating rather than none at all if possible.

"Benchmarks' may play into this last example also. If the 3.5 team from their section did well at nationals, then that would tend to drive up the ratings for all the 3.5s in that section.
Hmm... you'd have to put some "starting rating" for self-rates into the system wouldn't you? Or are you saying that a starting rating is generated after their first match? Possible but I doubt it. As for the benchnmark effect, I'd like to know the details on how that works exactly but those are hard to come by. The only thing I'd say is that benchmark adjustments could easily "degrade" the system as opposed to "equalizing" it, due to the weight given to an extremely small and possibly non-representative sample.

kylebarendrick
12-15-2010, 02:05 PM
I believe that the starting rating (their initial dynamic) is generated by their first match against a player with a dynamic rating. As an example, if they played someone rated 3.42 and won 6-4, 6-3, then they'd get an initial dynamic of something like 3.52 (assuming 0.05 per break).

ian2
12-15-2010, 02:06 PM
I believe that the starting rating (their initial dynamic) is generated by their first match against a player with a dynamic rating. As an example, if they played someone rated 3.42 and won 6-4, 6-3, then they'd get an initial dynamic of something like 3.52 (assuming 0.05 per break).
Very well might be the case, and would actually make sense...

dizzlmcwizzl
12-17-2010, 10:04 AM
Or are you saying that a starting rating is generated after their first match? Possible but I doubt it.

It is my understanding that this is exactly what happens.

ian2
12-17-2010, 10:21 AM
It is my understanding that this is exactly what happens.
Yes, I'm now convinced that it is exactly what happens... I was wrong to doubt it.

tennis4josh
12-17-2010, 10:47 AM
Here is the guy who started as 4.0S in year 2006. His record goes as follows:

2006 4.0(7/3) 7.5(4/4) 8.5(6/1)
2007 4.0(16/1) 7.5(7/4) 8.5(4/1) 8.0 (5/3)
2008 4.0(7/1) 7.5(3/1) 8.5(3/1) 8.0 (17/5) 6.5(12/0)
2009 4.0(6/3) 7.5(3/1) 8.5(10/0) 8.0 (12/6)
2010 4.0(13/2) 7.5(10/1) 8.5(9/0) 8.0 (14/1) 7.0(3/1) Total (49/5)

In 2010 4.0 league, he won 9 out 11 singles matches he played. In those 9 wins, 1 was default, 4 went to 3 sets, and one win was 6-0, 6-0 against a 4.0C guy. The two losses were 4-6,3-6 and 3-6,6-7. Both the guys he lost to got bumped to 4.5 for 2011 season.

amarone
12-17-2010, 11:14 AM
I believe that the starting rating (their initial dynamic) is generated by their first match against a player with a dynamic rating. As an example, if they played someone rated 3.42 and won 6-4, 6-3, then they'd get an initial dynamic of something like 3.52 (assuming 0.05 per break).
We were told in a presentation given by the district Director of Leagues that it was actually after the second match.

welcome2petrkordaland
12-17-2010, 12:10 PM
i've scanned a great deal of this thread and have seen some shannanigans this year for sure by players as well as some poorly calculated NTRP ratings.

1. using match scores as measuring sticks of player levels is inherently problematic:

-you got score controlling which is rampant by sandbaggers but almost impossible to pinpoint and or prove. i've seen this first hand and since match records don't carry as much weight as one would think they should, these guys get away with it.

-you also have superclose deuce game situations where the score simply misrepresents the qualitative/quantitative player differences. for example, 6-1, 6-2 with all deuce games. . . rare but not highly unusual either.

SOLUTION: very hard on a logistical level to maintain such stats, but could the numbers of deuce games per set or per match be factored in somehow? just thinking outside of the box here.


2. Placing more weight/value on playing up vs. playing at level despite consistently losing when playing up is also a poor measuring stick. I've seen first hand guys who played at a higher level than previous year and lost all matches but didn't get bumped back down.

3. another fallacy to me is undermining the significance of winning 3-set matches among "same level" players. Is this true? Are 3 set matches between two "same level" players basically a wash in terms of ascertaining any NTRP difference? That seems counterintuitive.

4. another issue that needs closer scrutiny is participation in college level sports. i'm aware of athletic backgrounds being considering, but more needs to be done, as college level athletes with at worst intermediate skill levels should be playing up as their rate of improvement are much steeper with less time.

this is all for now that i have cuz i gotta go back to work. just some thoughts.

BustedString
12-21-2010, 10:53 AM
We have a guy in our local leagues who went 1-23 as a 4.0 based upon his Mixed rating from last year. This year he got a 4.0 C rating and was denied his appeal. The only match he won was in a 6.5 Mixed Combo league with a 2.5 partner, and that match didn't even count towards his rating. I have bolded the matches that counted towards his Computer rating. Of note, he did not play strong opponents and none of his opponents rated up. He lost to 3.5 players as well. The computer is definitely skewing people upwards and I feel badly for this guy who continues to participate despite the ratings slap in the face from the USTA computer. When viewing this record, remember that every match was a loss, except that 6.5 3rd set TB match which didn't even count towards his rating anyway. I personally think if a few of those 0-6, 0-6 losses had involved at least a game, then maybe he would have rated down because the computer excludes double bagels from DTRP/NTRP calculation.


8/20/2010 6-1, 6-1 #2 Doubles 6.5
8/22/2010 5-7, 6-4, 1-0 #1 Doubles 6.5 The win!
8/29/2010 7-5, 6-4 #1 Doubles 6.5
9/17/2010 7-5, 7-5 #3 Doubles 6.5
5/5/2010 6-2, 6-0 #2 Doubles 4.0
5/12/2010 6-2, 6-1 #1 Doubles 4.0
5/26/2010 6-1, 6-0 #1 Doubles 4.0
6/16/2010 6-0, 6-0 #3 Doubles 4.0
7/28/2010 6-1, 6-1 #1 Doubles 4.0
8/4/2010 6-0, 6-0 #3 Doubles 4.0
10/12/2010 6-0, 6-0 #1 Doubles 4.0
10/25/2010 6-2, 6-0 #1 Doubles 4.0
11/2/2010 6-2, 6-2 #1 Doubles 4.0
11/16/2010 6-1, 6-2 #1 Doubles 4.0
11/23/2010 6-0, 6-0 #2 Doubles 4.0
11/29/2010 6-2, 6-2 #2 Doubles 4.0
12/14/2010 6-1, 6-2 #1 Doubles 4.0
5/23/2010 6-3, 6-2 #3 Doubles 7.0
6/11/2010 6-1, 6-0 #3 Doubles 7.0
6/20/2010 6-3, 6-2 #3 Doubles 7.0
6/27/2010 7-5, 6-4 #3 Doubles 7.0
7/25/2010 6-4, 7-6 #3 Doubles 7.0
8/1/2010 6-2, 6-1 #3 Doubles 7.0
8/10/2010 6-4, 6-4 #3 Doubles 7.0

ian2
12-21-2010, 02:35 PM
We have a guy in our local leagues who went 1-23 as a 4.0 based upon his Mixed rating from last year. This year he got a 4.0 C rating and was denied his appeal. The only match he won was in a 6.5 Mixed Combo league with a 2.5 partner, and that match didn't even count towards his rating. I have bolded the matches that counted towards his Computer rating. Of note, he did not play strong opponents and none of his opponents rated up. He lost to 3.5 players as well. The computer is definitely skewing people upwards and I feel badly for this guy who continues to participate despite the ratings slap in the face from the USTA computer. When viewing this record, remember that every match was a loss, except that 6.5 3rd set TB match which didn't even count towards his rating anyway. I personally think if a few of those 0-6, 0-6 losses had involved at least a game, then maybe he would have rated down because the computer excludes double bagels from DTRP/NTRP calculation.


8/20/2010 6-1, 6-1 #2 Doubles 6.5
8/22/2010 5-7, 6-4, 1-0 #1 Doubles 6.5 The win!
8/29/2010 7-5, 6-4 #1 Doubles 6.5
9/17/2010 7-5, 7-5 #3 Doubles 6.5
5/5/2010 6-2, 6-0 #2 Doubles 4.0
5/12/2010 6-2, 6-1 #1 Doubles 4.0
5/26/2010 6-1, 6-0 #1 Doubles 4.0
6/16/2010 6-0, 6-0 #3 Doubles 4.0
7/28/2010 6-1, 6-1 #1 Doubles 4.0
8/4/2010 6-0, 6-0 #3 Doubles 4.0
10/12/2010 6-0, 6-0 #1 Doubles 4.0
10/25/2010 6-2, 6-0 #1 Doubles 4.0
11/2/2010 6-2, 6-2 #1 Doubles 4.0
11/16/2010 6-1, 6-2 #1 Doubles 4.0
11/23/2010 6-0, 6-0 #2 Doubles 4.0
11/29/2010 6-2, 6-2 #2 Doubles 4.0
12/14/2010 6-1, 6-2 #1 Doubles 4.0
5/23/2010 6-3, 6-2 #3 Doubles 7.0
6/11/2010 6-1, 6-0 #3 Doubles 7.0
6/20/2010 6-3, 6-2 #3 Doubles 7.0
6/27/2010 7-5, 6-4 #3 Doubles 7.0
7/25/2010 6-4, 7-6 #3 Doubles 7.0
8/1/2010 6-2, 6-1 #3 Doubles 7.0
8/10/2010 6-4, 6-4 #3 Doubles 7.0

Ouch... 10char.

escii_35
12-22-2010, 12:42 PM
We have a guy in our local leagues who went 1-23

Going 1-23 and not seeing any movement ... ouch.

ronray43
12-22-2010, 05:10 PM
A 4.0 guy at our club went 23-1 in matches that count toward NTRP (all matches excluding mixed), including a 5-0 record at districts. Stayed a 4.0 . . . .geez, wish I could have a record like that and never get moved up!

doubleshack
12-22-2010, 05:48 PM
I can understand the 'logic' behind not counting 6-0,6-0 scores. It is hard to rate the difference between a 4.5 putting the beat down on a 4.0, vs a 4.5 putting the beat down on a 3.5. Both of those are 6-0, 6-0, but clearly, they have a different impact.

So, for the winner, understandable those scores don't count it. But for the loser, as BustedString has shown above, its tough that those get thrown out. Its clear the person in the example loves tennis, they keep plugging away. I hope things work out eventually. Too bad they are the anomaly in a system that has to be designed for to accomodate 300k+ people.