CiscoPC600
Hall of Fame
Have you ever done the same analysis with Schmke's ratings?
The dreaded EOY Rating thread
- Thread starter Brilliant
- Start date
Schmke tries to model/reverse engineer the USTA algorithm as accurately as possible because he is specifically trying to give people advice on USTA ratings. TR created their own, which is inferior but I guess simpler to calculate or something, but is not directly analogous to the USTA DNTRP.
I think the key question is: what will be your new dynamic rating after the new match result is factored in? The answer to that question will likely depend on more information than you have provided, namely the history of your current rating.
Your current 3.01 could be based on many prior results with a pretty stable rating trajectory. It could also be based on a single prior match result.
I agree with @J_R_B that Tennisrecord surely does not match NTRP, but I disagree that it's completely unreasonable. In the case that your current 3.01 was stable over several prior matches, TR says that your updated rating after this match would be 3.13. That seems pretty reasonable to me. A healthy boost from a result that was much, much better than expected, but not a crazy over-reaction. It's likely that NTRP's result would be different than 3.13, but could it be that much different?
In the case that you and your partner had no prior history and this was your first ever match result, TR would assign your initial dynamic rating at 3.49. That also seems reasonable to me. The only thing known about you is that you played in a 3.5-level match and destroyed two bottom-end players, so it's a reasonable starting point at the high end of the 3.5 range until more data come in.
It's not unreasonable for the example where everyone starts at 3.01. Their approach is very much unreasonable in other circumstances that I outlined.
Can you give an example where TR's updated dynamic rating (not the match rating) is unreasonable?
When schmke ran my rating last year it was .02 different than tennisrecord. I had played a lot of matches last year at the time of him running my rating. .02 differential is close enough for me and tells me the results of the calculations they are using are obviously similar. It also, like it or not, lends a certain degree of cred to TR and their process.
schmke
Legend
The dynamic ratings are based on the match ratings in the same way they are for the USTA. There is a buffer built in for that with the averaging. The match ratings can quite be unreasonable, though, and unnecessarily so.
The ratings will (or should) converge with higher match volume, if you are playing at a reasonably consistent level throughout the year. How the handle more sporadic play is a bigger differentiator. Schmke, since he is deliberately reverse engineering the actual USTA calc in a way that TR is not, does a much better job of actually getting the DNTRP (and resultant bumps and whatnot) correct.
TR is interesting, and I really appreciate that they are so transparent with a free product, but the calculation is definitely flawed. What they do right generally is rank players within a team. If you look at a team in TR (that you are familiar with), you can tell that the best players generally are near the top of the list, so that makes it a good tool for scouting opponents that you are not familiar with. As for using it to predict whether you will be bumped or not or more generally trying to discern you're rating on the DNTRP scale, it is virtually useless for that.
I think this is where you're making an assumption that may not be true. As far as we know the idea of a "match rating" is an invention of TR. I've never seen USTA mention anything called a "match rating" in their FAQ or other descriptions. They just say that they adjust a player's dynamic rating up or down based on the expected result vs. the actual result. That adjustment doesn't necessarily have to involve calculating a match rating. So as long as TR's end result of the adjustment is reasonably accurate, that's all that matters.
The only scenario when a match rating must be used by USTA is for a player's very first match. When there's no prior rating to adjust, there is only the one match score and the opponent's rating to use for the calculation. That's basically what TR's match rating is, and for that scenario I think they have a reasonable formula for calculating it.
The USTA uses the match rating for the dynamic rating for strike calculation for the first two matches for self-rates, then averages the current match rating with the last two or three dynamic ratings to calculate a current dynamic rating for subsequent matches. I think that's the same or very similar to what TR does.
Do you think TR's calculation formula for a self-rate's first dynamic rating is much different than USTA's?
No, not really. The ratings for self-rates would only differ if the algorithm differentials were significantly different (i.e. if USTA says a 6-2 6-3 win is worth a differential is 0.4 and TR says it's 0.1 or something). I don't know what the differentials are that either side uses, but I suspect there are only inconsequential differences between the two. The significant difference in the match rating algorithm comes into play when both sides have a dynamic rating and whether you anchor ratings to the opponent's rating exclusively or the average of the ratings. When only one side has a dynamic rating, you have to anchor the non-rated opponent's match rating to the rated opponent's dynamic rating and not change the rated opponent's dynamic rating at all because there is no information available about the non-rated opponent to use to evaluate the rated opponent's result.
The only exception to this is that the USTA will occasionally set self-rates to the center of their self-rated band for leagues that are essentially start-ups where a majority of the players are self-rated and it would take too long for enough matches involving rated players to accumulate so that everyone gets a valid rating by the end of the season. I've never seen this in practice, though. I suspect these are small leagues in remote areas where very few players have a rating (like consider the USTA creating an Alaska section - how would anyone who signs up the first year get a valid rating by the end of the season if no one in the section has an existing rating?).
OK, taking your prior example of pre-match 3.00 and 3.50 opponents playing to a tie in games.
Assuing the 3.00 and 3.50 were stable over a few prior updates for each opponent, TR would update their ratings to 3.12 and 3.38 after they tied each other. Ignoring the match rating and just looking at the final result of the update, is it really that unreasonable? I don't think that USTA's update for the same scenario would be substantially different.
The averaging in the dynamic calculation definitely tempers the stupidity of the match rating, but why even have a stupid match rating algorithm in the first place?
GrassCourtFan
Legend
Could be the main reason to have a match rating is to use it to calculate updated ratings for the opponent. It's just an intermediate step in the calculation they decided to reveal for whatever reason.
If they called it "skill adjustment factor" instead of "match rating" that would probably be a better name for that number, though.
As a number to be used in the averaging step, it makes the dynamic rating more responsive to a legit change in a player's skills.
Consider an extension to your example: say the 3.00 player played a different 3.50 opponent every week, and every result was an exact tie in games. Wouldn't you expect that the former 3.00 player's rating should converge close to 3.50 after a reasonable number of those results? They are consistentely and repeatedly showing that they are even with 3.50 players. To me, it's not at all stupid to assume that the player has legitimately raised his skill level, rather than assuming that every one of those opponents is playing worse than usual against this one guy.
Moon Shooter
Hall of Fame
I understand what jrb is saying and I think it’s a decent point about the match rating as it is. But I also think as long as it comes out in the wash somewhere in calculating the new dynamic rating it is fine.
I always looked at the “match rating” as sort of a “performance rating”. And generally I do think it makes sense to assume you have no prior rating in looking at your “performance rating”. In other words your rating going in to a event should not effect your “performance rating” at an event.
Let’s say player A and B play in a tournament. Now let’s say they do not play each other but they do play against the exact same 5 players and get the exact same scores against them. It would make sense to say they both had the same performance in that tournament and so their “performance rating” should be the same. It doesn’t seem right to say that since player A was rated higher going in to the tournament he actually performed better then player B in that tournament.
For people more into math the I am here is a website that talks about how performance ratings can be calculated in chess:
en.m.wikipedia.org
I always looked at the “match rating” as sort of a “performance rating”. And generally I do think it makes sense to assume you have no prior rating in looking at your “performance rating”. In other words your rating going in to a event should not effect your “performance rating” at an event.
Let’s say player A and B play in a tournament. Now let’s say they do not play each other but they do play against the exact same 5 players and get the exact same scores against them. It would make sense to say they both had the same performance in that tournament and so their “performance rating” should be the same. It doesn’t seem right to say that since player A was rated higher going in to the tournament he actually performed better then player B in that tournament.
For people more into math the I am here is a website that talks about how performance ratings can be calculated in chess:
Performance rating (chess) - Wikipedia
Moon Shooter
Hall of Fame
Whatever, tr does for self rates it is indefensible and leads to major problems for their rating system. I pointed out a few examples in the thread talking about the tr algo.
loveallcats
New User
If you do some online course on usta you can view them 3 days in advance!
travlerajm
Talk Tennis Guru
Can you give example of why you think TR’s match ratings are stupid?
In my experience, the algorithm appears very simple and reasonable. In fact I can almost reverse engineer it just from sampling a few of my own match ratings.
If I lose 8 games against an average 4.5c, it spits out a match rating of about 4.4. If I lose 6 games, match rating of about 4.5. If I lose 4 games, my match rating is about 4.6.
In other words, I need to average 6 games lost per match or less to manage my rating and not get bumped to 5.0 on TR.
GrassCourtFan
Legend
The match ratings end up strongly dependent on your opponents' rating, in a way that doesn't make sense when you consider both players ratings together.
For example, If a 4.0 plays a 4.5 and they get a "tie" match (for example 7-6 5-7 1-0), the 4.0 will get a match rating of 4.5 and the 4.5 will get a match rating of 4.0.
It makes sense on an individual level - "if I play someone with a rating of X and play exactly as well as them, I get a match rating of X" but is weird when you consider both of them together (for the same match, if two players play exactly the same, they can get wildly different match ratings, based on their opponent's rating.) You could argue that the thing that makes more sense would be for both players to get the same match rating, which would be exactly in the middle between the two players' initial ratings.
(I personally think this is fine, but I can see why some people don't like it.)
travlerajm
Talk Tennis Guru
It’s actually quite simple, with the match rating calculated separately for each player based on the opponent strength calculated from ratings of the other 3 players on the court.
Last edited:
It is simple, but if you think it's not stupid that two people who play to a "tie" (for rating purposes) can end up with wildly different ratings for that match, than we just disagree.
With enough match volume it evens out... but TR handles selfrates could be improved. There's a guy who was undefeated in men's 3.5 doubles and singles; his TR was something like 3.68. But this month, he decide to try Mixed doubles for the first time, and it started his mixed rating at 3.25... which effects how his opponents get rated, his partner gets a bigger bump up in rating b/c it looks like she's playing w/ a worse player... obviously mixed won't affect his dynamic rating or year-end bump b/c he has so much men's league matches, but it seems like the rating system for TR could be improved much more here. After 3 mixed matches, he's up to 3.49 so I'm sure w/more time it'd look better.
travlerajm
Talk Tennis Guru
As far as I know, when a player plays mixed for the first time, he/she starts from scratch with a new TR rating independent from the same gender rating. The first mixed match, he gets an S match rating, and the match result is used to calculate his initial mixed dynamic rating, while match result has no effect on the other 3 players’ ratings.
I suppose you can fault TR for not feeding additional available data into the calculation of the initial mixed rating, but you could say the same for other rating systems that keep track of multiple rating types separately.
travlerajm
Talk Tennis Guru
Can you provide an example, with TR dynamic ratings pre-match for all 4 players, and match ratings, that you think seems unfair?
If I’m a 4.4 and I play with 3.1 partner against a 3.8 and a 3.9, and the match ends in a tie:
Expected rating for me would be 3.8 + 3.9 - 3.1 = 4.6.
Expected rating for my partner = 3.8 + 3.9 - 4.4 = 3.3.
Expected rating for 3.8 opponent = 4.4 + 3.1 - 3.9 = 3.6.
Expected rating for 3.9 opponent = 4.4 + 3.1 - 3.8 = 3.7.
This is generally how most rating systems work. Does this look unfair to you?
time_fly
Hall of Fame
The problem with the TR algorithm is the handling of self-rates. When I looked at the data, I was surprised at the number of USTA doubles matches in which at least one of the players has played fewer than 3 matches, even at the 3.5 level. Those matches basically don’t count for the established players and serve only to help set an initial rating for the new player. But that initial rating is often very inaccurate because there’s an outlier match among the first 3. So now you have a situation where many of an established player’s matches “don’t count” and an inaccurate number pops out for the new player which starts affecting everyone after that. The only reasonable way to handle this is to propagate ratings backwards as well as forwards. I think the end of year rating calculation may try to do this, but the transparent dynamic rating is all based on forward calculation.
travlerajm
Talk Tennis Guru
My most recent match was handled very differently by WTN than by TR.
My opponents had very strong WTN ratings with my partner relatively weak on WTN. Opponents had strong TR ratings in same gender play, but relatively weak mixed ratings based on small sample size.
I won the match in a super tiebreak, but lost more games than opponent.
WTN only counts sets, so it gave me strong match rating and my dynamic rating improved.
TR only counts games without paying attention to who wins. So TR gave me a crummy match rating and my TR mixed dynamic rating went down.
It's only stupid if the match rating is intended to be interpreted as an absolute estimate of how well each opponent played in that match, accounting for both of their past histories combined. If a pre-match 3.00 "ties" a pre-match 3.50, then I agree that it's weird to assume both that the underdog played like a 3.50 and the favorite played like a 3.00 that day. If that had happened, then it would not have been a tie, the underdog would have crushed the favorite. If the match rating was defined the way you want, then instead both players would get a match rating of 3.25.
But that was never the intention of TR's match rating. Their definition is, for each player in turn: if we knew nothing about you other than this match result and the opponent's prior rating, what rating would you get? If someone ties a 3.00, they get a 3.00 match rating. If someone ties a 3.50, they get a 3.50 match rating. Perfectly logical under that definition.
I don't think either definition is more or less stupid than the other, they're just different. I personally like scanning through my match ratings on TR, and seeing how they would rate each result in isolation as if it had been my first match. There is value in looking at them from that perspective.
Moon Shooter
Hall of Fame
Unless the volume of matches includes other self rates. Then it tends to really go off the rails.
Here are a few of the unexplained self rate issues:
TR calculations
This is guy from our club who decided to figure out TennisRecord calculations. Might be interesting to some of you... https://www.youtube.com/@RajeevGoelTennis
tt.tennis-warehouse.com
TR calculations
I don't know where I got the impression ntrp leans heavily on match winning %, whereas UTR reflects more games won. It seems to be games won that matters for NTRP too, although it can easily be outweighed by quality of opposition and partner, like the TR calculations show. Those other factors...
tt.tennis-warehouse.com
TR calculations
I don't know where I got the impression ntrp leans heavily on match winning %, whereas UTR reflects more games won. It seems to be games won that matters for NTRP too, although it can easily be outweighed by quality of opposition and partner, like the TR calculations show. Those other factors...
tt.tennis-warehouse.com
silverwyvern4
Semi-Pro
And if the 3.00 and 3.5 keep playing and tieing, they will eventually both end up tied rating at 3.25
Icsa
Semi-Pro
A match win should never make your dynamic rating go down. Even if according to the previous rating, the match "should" have been won with a higher won/lost game ratio, it still doesn't make any sense to give a negative rating to a match win. For this reason alone, TR is bogus.
silverwyvern4
Semi-Pro
Tell that to USTA
From the computing blog:
Does it not matter if I win or lose then?
A: Strictly speaking, yes. It is possible that the expected result is that you will lose 6-2,6-2 and so if you lose 6-4,6-4, you did better than expected and your rating will go up. Similarly, you could be expected to win 6-2,6-2 and instead win 6-4,6-4 and your rating could go down.
Does it not matter if I win or lose then?
A: Strictly speaking, yes. It is possible that the expected result is that you will lose 6-2,6-2 and so if you lose 6-4,6-4, you did better than expected and your rating will go up. Similarly, you could be expected to win 6-2,6-2 and instead win 6-4,6-4 and your rating could go down.
Moon Shooter
Hall of Fame
Accuracy is one of the most important aspects of a rating system. So if it is more accurate by counting games won and lost instead of just matches won and lost then I am in favor of counting games.
WTN only counts sets. I think the problem with only counting matches is the system would be starved for data and therefore less accurate. I suspect that is even true for WTN.
GrassCourtFan
Legend
Why not?
If I'm according to a computer a 4.0, and I pull out a tight nailbiting win over a 3.0... ...that's information for the rating system that I'm pretty close to 3.0 and not 4.0...
Moon Shooter
Hall of Fame
I am not sure but I thought your ATP does not go down even if you lose matches. So you just want to get invited and play in as many matches that are potentially worth more points as you can. It would be a horrible system for rating rec tennis players.
GrassCourtFan
Legend
None.
Big difference is that the the ATP requires players to play at least 19 tournaments a year, and gives big fat zeros in the rankings for missing that number of tournaments, but any rec system needs to work decently for the sorts of players who play a single-digit number of ranked matches in a year (as low as 3), without creating a system that just rewards playing more and more.
travlerajm
Talk Tennis Guru
UTR also lowers your rating if you win with a margin below the expected result. I believe, but am not certain, that WTN uses an ELO algorithm that works more like you are saying you’d prefer, where a win always improves your rating.
Nate Silver used that type of ELO system when he established the now-defunct sports ratings page of 538.
Icsa
Semi-Pro
It takes the fun out of the recreational matches (USTA or not) if one needs to do everything to win by 6-0, 6-1. If I have a big lead like 5-1 or 5-2, I just take it easy and have fun knowing that the win is secured. Sometimes I even have some pity if I'm about to win by 6-0 and let some games for the other side so they don't feel crushed. Again, recreation is about having fun and get out and exercise.
If they want to give me zero credit for a win, fine. But rating going down for wins just doesn't seem right.
If they want to give me zero credit for a win, fine. But rating going down for wins just doesn't seem right.
Last edited:
I noticed this for a singles match (since it's easier to see the math with just two people) - S2 in this match.
TennisRecord.com - Estimated Tennis Ratings & Stats
Lookup adult and junior tennis players and stats.
www.tennisrecord.com
For this match, JR got a match rating of 3.51 and MJ got a rating of 3.33 despite the match being an 8-8 "tie" in games. That's what is not fair.
What else is a match rating for? If it's something else, that's what stupid.
travlerajm
Talk Tennis Guru
By giving extra credit to the player who exceeds expectations, the rating algorithm can be more responsive and less sluggish. In rec level play, player level can fluctuate a lot due to injuries and how busy people are at work, so it’s important for the algo to be responsive.I noticed this for a singles match (since it's easier to see the math with just two people) - S2 in this match.
TennisRecord.com - Estimated Tennis Ratings & Stats
Lookup adult and junior tennis players and stats.
For this match, JR got a match rating of 3.51 and MJ got a rating of 3.33 despite the match being an 8-8 "tie" in games. That's what is not fair.
First, the match rating is for giving someone an initial dynamic rating when they don't have a prior rating. I think you've agreed that TR's version works fine for that purpose.
Second, the match rating is for updating the dynamic rating. I think you've agreed that TR's version works fine for that purpose too.
Those are the most important functions of the match rating. If you want it to also have some other particular meaning in isolation, then that's more a matter of your taste and not a logical flaw of TR.
schmke
Legend
There are ways to be more responsive than giving extra credit. That is a hack that has other side-effects.
travlerajm
Talk Tennis Guru
I believe TR is simply calculating each of the two players’ match ratings separately.
If the two played to a tie, the stronger rated player underperformed by playing to the pre-match level of the weaker player.
The weaker rated player overperformed and played to the pre-match level of the stronger player.
This isn’t really a hack, but a quite logical way to calculate the match rating. Otherwise, it would take a long time with many ties in a row for the weaker rated player to catch up.
It’s not really extra credit, but rather “full” credit.
If a pre-match 3.00 "ties" a pre-match 3.50, TR will bump the 3.00's dynamic rating up to about 3.12. Is that too much credit for the overperformance? You probably know the answer but won't tell, haha. It seems reasonable to me, but who knows.
I'm not really defending TR, I just think that the problems they have come from other things aside from the way they use match ratings. Like including the wrong or not enough matches in their calculations, or their strange between-season adjustments, or the way they inexplicably seem to not follow their own formula in many instances.
Similar threads
