End of year ratings question

ATX Tennis

New User
Sorry if this is readily accessible...I couldn’t find it. When is the last date that this year’s matches will count towards end of year ratings? Usually it’s some date in mid-late October. Thanks!
 

texacali

Rookie
Fearing end of year...do not want to move up. Curious. If your team qualifies for nationals, do folks tend to get moved up?
 

schmke

Hall of Fame
Fearing end of year...do not want to move up. Curious. If your team qualifies for nationals, do folks tend to get moved up?
Depends how you do, depends how your team does.

Even teams that make semis and/or win don't have every player bumped up, but Nationals team members certainly do get bumped up at a higher rate than the rest of the players.
 

texacali

Rookie
Depends how you do, depends how your team does.

Even teams that make semis and/or win don't have every player bumped up, but Nationals team members certainly do get bumped up at a higher rate than the rest of the players.
Thanks. Guess we'll see in a couple of weeks.
 

ShaunS

Semi-Pro
Thanks. Guess we'll see in a couple of weeks.
If you just can't wait, I know a guy who does estimated DNTRP reports that are pretty reliable:

They're actually useful for a lot more than simply your dynamic rating, and I thought what I received in the past was well worth the cost.
 

Tiafoe

Rookie
Who actually designates who is bumped up or down? Is it admins at National level, Sectional level or district level?
 

OnTheLine

Hall of Fame
Who actually designates who is bumped up or down? Is it admins at National level, Sectional level or district level?
It is a computer. Basically an algorithm that spits out a number.
At the end of the year at the national level there may be some tweaks where adjustments are made based on how certain sections performed at nationals.

(e.g. if your particular section performed very well in post season play the ratings of that section in general may be pushed a little higher, if the section as a whole did poorly post season then the entire section may get a downward push)

But it is not an actual human looking at an individual record and making a determination, except in perhaps some rare cases (like people who stood out in post season play and likely to get double-bumped)
 

CiscoPC600

Professional
Haven't seen as much talk this year about the end of yer ratings update. If last year is any indication, aren't we a week or so away from the end of year rating change?
 

MaxTennis

Rookie
@schmke I think I may have asked you this before, but do Open tournament results count toward NTRP?

I have a win over a guy who is about 4.80 DNTRP and was 10.72 UTR in an Open tournament.

Since it isn't reflected on TennisRecord, my DNTRP is still about 4.52 and my UTR is now 10.51.
 

tpro2000

Rookie
I was told 12/2 this year but not 100% sure of the validity. Either way it's always after Thanksgiving weekend
 

schmke

Hall of Fame
@schmke I think I may have asked you this before, but do Open tournament results count toward NTRP?

I have a win over a guy who is about 4.80 DNTRP and was 10.72 UTR in an Open tournament.

Since it isn't reflected on TennisRecord, my DNTRP is still about 4.52 and my UTR is now 10.51.
I don't think Open tournaments count, although I think it may be at a section's discretion.
 
Last edited:

schmke

Hall of Fame
I was told 12/2 this year but not 100% sure of the validity. Either way it's always after Thanksgiving weekend
Yeah, it is generally safe to assume the Monday after Thanksgiving, sometimes it stretches into the Tuesday or Wednesday depending on when Thanksgiving falls and if the USTA gets it done in time.
 

MaxTennis

Rookie
Yeah, it is generally safe to assume the Monday after Thanksgiving, sometimes it stretches into the Tuesday or Wednesday depending on when Thanksgiving falls and if the USTA gets it done in time.
I think you quoted the wrong post ;)
 

winchestervatennis

Hall of Fame
It is a computer. Basically an algorithm that spits out a number.
At the end of the year at the national level there may be some tweaks where adjustments are made based on how certain sections performed at nationals.

(e.g. if your particular section performed very well in post season play the ratings of that section in general may be pushed a little higher, if the section as a whole did poorly post season then the entire section may get a downward push)

But it is not an actual human looking at an individual record and making a determination, except in perhaps some rare cases (like people who stood out in post season play and likely to get double-bumped)
I get what you’re saying, but if it were that simple, the ratings should be ready by today since it looks like all nationals are over but mixed. They’ve been doing it long enough that whatever adjustments are made for the sections that do well or poorly can easily be run through a computer calculation. So why would it take 3 more weeks if some level of human looking at them doesn’t take place. Methinks lists are shared at some level, maybe local, for input.
 

schmke

Hall of Fame
I get what you’re saying, but if it were that simple, the ratings should be ready by today since it looks like all nationals are over but mixed. They’ve been doing it long enough that whatever adjustments are made for the sections that do well or poorly can easily be run through a computer calculation. So why would it take 3 more weeks if some level of human looking at them doesn’t take place. Methinks lists are shared at some level, maybe local, for input.
Well, I do have my year-end calculations done! Preliminary ones anyway.

So yeah, the USTA "should" be able to do it sooner too, but I think they have some built in review cycles to make sure nothing has run amuck, and I think there is some opportunity for regional adjustments after looking at what the computer spits out. Certainly some years there have been a lot of bumps up over all or in different regions that I think could only be explained by manual review and adjustment.
 

ej28405

New User
Well, I do have my year-end calculations done! Preliminary ones anyway.

So yeah, the USTA "should" be able to do it sooner too, but I think they have some built in review cycles to make sure nothing has run amuck, and I think there is some opportunity for regional adjustments after looking at what the computer spits out. Certainly some years there have been a lot of bumps up over all or in different regions that I think could only be explained by manual review and adjustment.
Just curious... Does mixed or combo count towards year end rating? And what about tournament results over the next couple weeks? Do they get factored into this or next years rating? Thanks!
 

schmke

Hall of Fame
Just curious... Does mixed or combo count towards year end rating? And what about tournament results over the next couple weeks? Do they get factored into this or next years rating? Thanks!
Mixed only counts if you play only Mixed. Otherwise it is just your Adult league matches (and those leagues that your section opts in to including). Combo almost never counts, although again, it is something a section can opt to include. I think tournaments played after 11/10 won't count for 2019.
 

Vox Rationis

Semi-Pro
@schmke If I remember correctly you have to have 3 postseason matches for those matches to receive extra weight in your ratings calculation? Is that still the case? If you only have two rated postseason matches, do they treat it just like your regular season matches?
 

leech

Rookie
@schmke If I remember correctly you have to have 3 postseason matches for those matches to receive extra weight in your ratings calculation? Is that still the case? If you only have two rated postseason matches, do they treat it just like your regular season matches?
This is from an older document but provides some useful guidance on how postseason matches are weighted (20% for 1 postseason match, 40% for 2 postseason matches, 50% for 3 or more postseason matches): http://assets.usta.com/assets/558/Microsites/dps/usta_master/usta/doc/content/doc_90_106.pdf
 

Vox Rationis

Semi-Pro

leech

Rookie
More info on how the Nationals Benchmark ratings were used to calibrate ratings across Sections. From 12 years ago, so much could have changed -- I don't think they use verifiers anymore, but do think they assign greater weight to the rating of those who play the semis and finals at Nationals.


BENCHMARK PLAYER RATINGS (how are they determined?)
By: Bob Greene USPTA, USTA Certified Verifier & Chairman of USTA NTRP Computer Sub-Committee


Annually each Section of the USTA will send up to twenty teams at all levels and genders to USA League National Championship Events. Those teams will play against each other in four flights of four or five teams in each flight. The draw is done at random. One event may be Florida, Texas, New England and Hawaii ... the next Eastern, Southern, ******* and Northern California.

Before and during these matches, no less than four of the most experienced NTRP Verifiers from different areas of the country will research the players match result history, multi-year rating history and player profile information. They will then observe the players competing against several different teams over a period of three days. All match results are entered into the NTRP Computer during the events.

The Verifiers are observing and are more specifically looking for lopsided match results, disparity of level between doubles partners and player improvement over the course of a season. The players who emerge from their respective flights to the semifinal and final rounds are given "absolute ratings". That rating is a number that is static for the purpose of comparison against other players.

These "Benchmark Ratings" are entered into the NTRP Computer and the computer program is run calculating ratings for all of the players who competed at the event. Although the NTRP Computer has an excellent track record of being correct, the National Verifiers makes a few adjustments based on reasons stated above. All of these players are National Benchmarks and their Ratings are deemed not changeable by regulation.

These National Benchmark Ratings are entered into the NTRP Computer and will filter down in each respective USTA Section and all players competing in NTRP audited and regulated venues will receive a rating if they played two or more matches. The primary goal and purpose of this methodology is to create and maintain uniformity in ratings on a nationwide basis. No matter what the picture appears to be from the bottom looking up, it is painted from the top down.
 

Doan

Rookie
I believe elements of it are still accurate, but it was over 10 years ago and some things have changed.
If the DQ zone has stayed the same I'm surprised at the actual # of DQs that happens. For a 3.0S > 3.30 and 3.5S >3.80.
 

leech

Rookie
Fearing end of year...do not want to move up. Curious. If your team qualifies for nationals, do folks tend to get moved up?
Here is some analysis I shared with my 3.5 ladies team this week, based off of TennisRecord data (which not only is wildly inaccurate, but has been down for the last couple of days!):

In anticipation of the new ratings for 2020 (expected to be released Dec 2), I took a look at the 2018 40+ 3.5W National Champion team to see how many of them got bumped for the following season. The champs from PacNW had 18 ladies on the roster, including a couple of 3.0 players.

All bumped to 4.0 for 2019, except players with these records and TR ratings:

14-8; 3.30
11-5; 3.29
17-12; 3.23
11-5; 3.17
9-3; 2.95
1-1; 2.79
1-5; no rating
3-4; no rating

Everyone above 3.30 bumped, plus a 3.24 lady (who was just 14-6, 5-4 in postseason). Overall, just 10 of 18 bumped to 4.0.

If 2018 is any indication, then everyone on our team that is rated 3.30 and lower may be safe from bump up:

30-28; 3.30
36-11; 3.28
41-26; 3.23
36-32; 3.20
6-2; 3.19
64-26; 3.16
11-4; 3.07

Those of you that have estimated TR ratings above 3.30 should brace yourselves for the possibility of getting bumped....

Perhaps 2018 was an anomaly for 3.5 bump ups in MidAtlantic. 19 of the 20 guys on my 40+ 3.5 Nationals-winning team got bumped, including players with these TR stats:

3.28; 72-39 (2-4 in postseason)
3.28; 44-23 (2-0 in postseason)
3.32: 65-27 (2-1 in postseason)
3.32; 27-16 (didn’t play postseason)
3.36; 102-45 (3-3 in postseason)
3.39; 51-43 (1-0 in postseason)

And the 2018 18+ 3.5W from MidAtlantic went just 1-3 at Nationals, but had all but four 3.5 ladies bump up (and two of them appealed UP anyway), including ladies with these TR stats:

3.29; 97-49 (6-4 in postseason)
3.33; 37-16 (1-0 in postseason)
3.37; 105-37 (7-3 in postseason)
3.40; 56-34 (4-2 in postseason)

Seems they really gave a lot of weight to making Nationals, at least in 2018 among the MidAtlantic 3.5 population.
 

denoted

Rookie
I would love to know the cause of the error in Tennis Record and TLS (and maybe, to a lesser extent, in @schmke's ratings). The algorithm itself seems fairly straightforward, so the errors must result in playoff adjustments. But can that explain it all?
 

leech

Rookie
I would love to know the cause of the error in Tennis Record and TLS (and maybe, to a lesser extent, in @schmke's ratings). The algorithm itself seems fairly straightforward, so the errors must result in playoff adjustments. But can that explain it all?
My hunches:
(1) TR ratings are vastly off from @schmke's, and presumably, the actual dynamic ratings as calculated by the USTA. It isn't unusual for these estimates to be off by 0.25 to 0.40, which is huge spread!
(2) The TR ratings do not assign greater weight to postseason matches, as the USTA does (I believe @schmke does, if you order a year-end report from him)
(3) No one knows the additional weight the USTA assigns for Nationals semis/finals participants, and the resultant trickle-down effect.
(4) I suspect manual adjustments can be made at the Section level (just not to those that have played at Nationals semis/finals...seems those are "absolute" ratings that cannot be manually adjusted).
 

Max G.

Legend
The errors in TLS and TR are easy to explain - they’re not actually running the NTRP algorithm, they’ve made up their own algorithms that give similar looking numbers but not actually the real thing.
 

denoted

Rookie
Is TR error distributed randomly? From your (@leech's) earlier comment, it seemed like the errors were all underrating (playoff results). TLS looks much lower on average than TR from what I can see, which would mean that it's even more inaccurate?
 

denoted

Rookie
The document posted above says that a .29 difference equals 4 games. Another USTA document says that the expected score between 3.01 and 3.49 is 12 games. Isn't that enough data to estimate the (non-playoff adjusted) algorithm? It's just that and rolling averages, right? I'm sure the expected .48 spread is tighter at the marginal levels, but that can't be the source of all the error.
 

Ft.S

Semi-Pro
Is TR error distributed randomly? From your (@leech's) earlier comment, it seemed like the errors were all underrating (playoff results). TLS looks much lower on average than TR from what I can see, which would mean that it's even more inaccurate?
I do not think the issue with TR ratings is limited to the weight of post-season matches. Throughout the year, once my # of rated matches goes beyond 12, the difference in rating results, compared to the @schmke reports I receive once a quarter is stark. Only at the end of September their ratings matched after about 55+ matches.
I do not know where TR is differing, but its ratings are very misleading IMHO. Possibly their comparison tables are off a bit, but the cumulative effect on rating calculations is significant.
 

leech

Rookie
The document posted above says that a .29 difference equals 4 games. Another USTA document says that the expected score between 3.01 and 3.49 is 12 games. Isn't that enough data to estimate the (non-playoff adjusted) algorithm? It's just that and rolling averages, right? I'm sure the expected .48 spread is tighter at the marginal levels, but that can't be the source of all the error.
I agree, the algorithm should be fairly easy to replicate... IF one had the starting ratings for all players. But because this (hugely significant) data point is not made available, the sites and others cannot be accurate. Over time, it's possible to hone in closer to the actual dynamic ratings by looking at players on the fringes, to see which players actually bumped or not. But the postseason and particularly the Nationals weighting make it impossible for these sites to be accurate, agree?
 

schmke

Hall of Fame
My hunches:
(1) TR ratings are vastly off from @schmke's, and presumably, the actual dynamic ratings as calculated by the USTA. It isn't unusual for these estimates to be off by 0.25 to 0.40, which is huge spread!
(2) The TR ratings do not assign greater weight to postseason matches, as the USTA does (I believe @schmke does, if you order a year-end report from him)
(3) No one knows the additional weight the USTA assigns for Nationals semis/finals participants, and the resultant trickle-down effect.
(4) I suspect manual adjustments can be made at the Section level (just not to those that have played at Nationals semis/finals...seems those are "absolute" ratings that cannot be manually adjusted).
I do incorporate year-end calculations and refine them more each year, with continued improvements in my accuracy. My ratings are still not perfect, but my experience and that of others that I hear from and many that post here, is that they are more accurate than the alternatives.

@leech hits the right things, notably #4. In some years there are clearly regional adjustments made and there is no way for an algorithm to proactively predict that. How this is done and filters throughout the section/district/area can vary from year to year. And this complicates the following year as getting accurate starting ratings is important to calculating accurate ratings in the new year. Which is in itself a big factor in why a system may not be accurate. Inaccuracies in year X automatically make it hard to be accurate in year X+1 as you don't have the right starting ratings. So a less accurate algorithm will continue to be less accurate for this reason alone.
 

BallBag

Semi-Pro
Other fun TR stats for my area.

4.5 to 5.0 – 3 Bump ups and 15 bump downs
4.0 to 4.5 – 1 bump ups and 19 bump downs
3.5 to 4.0 – 7 bump ups and 50 bump downs

TR doesn't think very highly of us.
 

ChaelAZ

Legend
Should be interesting this year for me. Was looking at TLS and Tennis Record (neither of which was working well) and last offical rating I had was from 2017. Dynamic was in the crapper and playing back at 4.0 has been a less than stellar comeback, but decent.
 

schmke

Hall of Fame
Other fun TR stats for my area.

4.5 to 5.0 – 3 Bump ups and 15 bump downs
4.0 to 4.5 – 1 bump ups and 19 bump downs
3.5 to 4.0 – 7 bump ups and 50 bump downs

TR doesn't think very highly of us.
Oaf. This sort of bump up/down rate would be wildly inconsistent with normal years in general. Either players in your area played very poorly this year, or TR is a tad inaccurate for your area. Few bump ups is not uncommon at higher levels, but only 7 up and 50 down at 3.5 is highly unusual. Who knows, perhaps everyone is managing their rating looking to form a super team at 3.0 and 3.5 next year ;-)
 

BallBag

Semi-Pro
Oaf. This sort of bump up/down rate would be wildly inconsistent with normal years in general. Either players in your area played very poorly this year, or TR is a tad inaccurate for your area. Few bump ups is not uncommon at higher levels, but only 7 up and 50 down at 3.5 is highly unusual. Who knows, perhaps everyone is managing their rating looking to form a super team at 3.0 and 3.5 next year ;-)
I looked at another area and it was a pretty even distribution so its not their algorithm. I wonder if were gonna get an above average number of bump downs this year.
 

Doan

Rookie
Disregarded A and S rated players - just looking at 2019 C rated Male players I see this for TR estimated ratings in my district. No way that many 4.5/4.0 guys are getting bumped down in Mid-Atlantic. Especially after the 18+ 4.5 guys won nationals.

4.5C - up=2 down=18
4.0C - up=3 down=26
3.5C - up=10 down=13
 

texacali

Rookie
I see TR is updated because my Natl matches from last weekend are in....ranking not yet calculated. Anxious to see that.


Here is some analysis I shared with my 3.5 ladies team this week, based off of TennisRecord data (which not only is wildly inaccurate, but has been down for the last couple of days!):

In anticipation of the new ratings for 2020 (expected to be released Dec 2), I took a look at the 2018 40+ 3.5W National Champion team to see how many of them got bumped for the following season. The champs from PacNW had 18 ladies on the roster, including a couple of 3.0 players.

All bumped to 4.0 for 2019, except players with these records and TR ratings:

14-8; 3.30
11-5; 3.29
17-12; 3.23
11-5; 3.17
9-3; 2.95
1-1; 2.79
1-5; no rating
3-4; no rating

Everyone above 3.30 bumped, plus a 3.24 lady (who was just 14-6, 5-4 in postseason). Overall, just 10 of 18 bumped to 4.0.

If 2018 is any indication, then everyone on our team that is rated 3.30 and lower may be safe from bump up:

30-28; 3.30
36-11; 3.28
41-26; 3.23
36-32; 3.20
6-2; 3.19
64-26; 3.16
11-4; 3.07

Those of you that have estimated TR ratings above 3.30 should brace yourselves for the possibility of getting bumped....

Perhaps 2018 was an anomaly for 3.5 bump ups in MidAtlantic. 19 of the 20 guys on my 40+ 3.5 Nationals-winning team got bumped, including players with these TR stats:

3.28; 72-39 (2-4 in postseason)
3.28; 44-23 (2-0 in postseason)
3.32: 65-27 (2-1 in postseason)
3.32; 27-16 (didn’t play postseason)
3.36; 102-45 (3-3 in postseason)
3.39; 51-43 (1-0 in postseason)

And the 2018 18+ 3.5W from MidAtlantic went just 1-3 at Nationals, but had all but four 3.5 ladies bump up (and two of them appealed UP anyway), including ladies with these TR stats:

3.29; 97-49 (6-4 in postseason)
3.33; 37-16 (1-0 in postseason)
3.37; 105-37 (7-3 in postseason)
3.40; 56-34 (4-2 in postseason)

Seems they really gave a lot of weight to making Nationals, at least in 2018 among the MidAtlantic 3.5 population.
 
Top