Swing Vision AI line calls and Analysis
- Thread starter Badmrfrosty
- Start date
swupnil
Rookie
@Mungo I would use Doubles if you're playing Australian doubles. I'm not sure how well it will work but certainly will be better than using a Singles mode. We don't support score overlays in Game by Game mode yet. It's on the roadmap though!
@Addxyz in our experience, yes the M1 is blazing fast and even faster than a MacBook Pro with a high end Intel chip because of the Neural Engine.
@Addxyz in our experience, yes the M1 is blazing fast and even faster than a MacBook Pro with a high end Intel chip because of the Neural Engine.
Nostradamus
Bionic Poster
does this thing work ?? is it finally ready to put down all the cheaters in USTA and league tennis ?
I originally got this for help with line calling but IMO it doesn’t seem mature for that, just yet. Among other things is too unreliable in terms of knowing which shot I am asking about (the close one of course!). Because the watch display is so small it is so critical that it knows what you’re asking for help with. I have some confidence this feature will get much better and they’re on a path to delivering that. That said, the other features of the service have been a wonderful surprise. I have been able to spot numerous flaws in my gameplay / service using this tool. And it gives my playmates a great laugh to see our match highlights shared. So I am more than satisfied.
CosmosMpower
Hall of Fame
Anyway to allow it to process the file while leaving the screen so it runs in the background? I hate how the app holds my phone hostage for 20 mins during processing.
swupnil
Rookie
@Nostradamus not ready yet. Still more work to do to get the accuracy and (perhaps just as important) experience right.
@CosmosMpower sadly this isn’t supported by iOS. Only option is to get a newer device (that can process in real-time), another iPhone/iPad that you’re ok surrendering, or getting the Mac app (which CAN process videos in the background).
@CosmosMpower sadly this isn’t supported by iOS. Only option is to get a newer device (that can process in real-time), another iPhone/iPad that you’re ok surrendering, or getting the Mac app (which CAN process videos in the background).
CosmosMpower
Hall of Fame
Hrmm I'm already on 12Pro. I know the ios can run multiple apps at the same time, not sure about video processing.
swupnil
Rookie
@CosmosMpower ios pretty much suspends apps when you exit them. There are some exceptions with background processes but those are limited to things like networking (which is how we upload videos to the cloud in the background). Running the A.I. is super intensive and isn’t allowed in the background.
With that said, with an iPhone 12 Pro you should definitely be able to process the video in real-time. That’s super surprising that it’s not happening. If you can DM me your username, we can take a look at your recent sessions to see what might be going on.
With that said, with an iPhone 12 Pro you should definitely be able to process the video in real-time. That’s super surprising that it’s not happening. If you can DM me your username, we can take a look at your recent sessions to see what might be going on.
CosmosMpower
Hall of Fame
After I end the session it shows me a bunch of tiles of players and asks which one is me. I usually just click a few and move on. After that screen it shows me the video and I can play it in the app but it's when I want to export to my phone that it's taking a long time to process in order to be able to save to my photos on my phone.
swupnil
Rookie
@CosmosMpower oh my bad! You’re just talking about exporting the video, not the initial A.I. processing. Yea sadly we have no way around that at the moment.
We are building out a new sharing feature that will let you generate a unique link for your highlights and play it right in the browser on any device. Maybe that would help so you don’t have to export in the first place?
We are building out a new sharing feature that will let you generate a unique link for your highlights and play it right in the browser on any device. Maybe that would help so you don’t have to export in the first place?
When Editing / Reviewing video, how exactly does the “include in highlight reel” function work? Let’s say I haven’t filtered out anything and I am in rally mode, and everything by default is ”included in highlight reel”. Will unchecking certain rallies mean that if I were to export at that time, those unchecked rallies would be excluded from the file download? Does that only work right then and there, or if I export later on, would the tool remember I had unchecked some rallies and saved that preference?
LegionFHDTL
New User
So I recently subscribed and things have been working great except for the exporting. I am on a M1 Macbook and can't seem to export the video with overlays?
Also I am using 1080p video for now but are there any plans to allow for import of 4k/30 or 4k/60 videos? I'm assuming the M1 chip can still process this well
Also I am using 1080p video for now but are there any plans to allow for import of 4k/30 or 4k/60 videos? I'm assuming the M1 chip can still process this well
swupnil
Rookie
@LegionFHDTL unfortunately we can’t export with overlays since there’s an issue with the video APIs on macOS. We need to just wait for this to get resolved in a future macOS.
We hope to support 4k in the future. The bigger issue is that to keep the complexity of the App down, we need the same algorithms to be able to run in real time on iPhone. Secondarily, backing up a 4K file to the cloud will also just take forever. We’re also seeing that beyond 1080p, the accuracy doesn’t improve all that much (though it does help for the far side of the court).
We hope to support 4k in the future. The bigger issue is that to keep the complexity of the App down, we need the same algorithms to be able to run in real time on iPhone. Secondarily, backing up a 4K file to the cloud will also just take forever. We’re also seeing that beyond 1080p, the accuracy doesn’t improve all that much (though it does help for the far side of the court).
LegionFHDTL
New User
Thanks for the update, hopefully mac OS fixes in future as overlays are amazing
I understand, forgot that it also has to run real time on iPhone as well.
Do you know why the files need to be backed up to the cloud if the processing is done on device? As long as the user has storage, shouldn't the user have option of forgoing back up to the cloud?
swupnil
Rookie
@LegionFHDTL they don’t have to, I suppose. It’s more again to just keep things simple and not open up the app to bugs where things accidentally DONT upload. But certainly seems reasonable to add this eventually!
LegionFHDTL
New User
Got it, thanks. Yeah I can imagine keeping it simple. Looking forward to every update in the future!
Sorry if this has been addressed, I attempted a search through the thread with no luck. I used the audio guided set up this weekend and it seemed to work pretty well for me. If I try to use my watch to setup the camera, it seems like it can take minutes to actually sync to my watch app so the audio set up was helpful. (I have an apple watch 3 and iphone SE.) My questions are - if I turn my phone on airplane mode, can I still use my watch to view the camera position for set up? And is there any way to correct the delay in syncing between the phone and the watch for the set up process (or is this because of my slower devices)? Thanks!
derick232
Rookie
I would imagine as long as you can maintain connectivity with your watch then yes. I believe early on they actually recommended putting your phone in airplane mode while recording to save battery life. That was probably a year or more ago and many updates have come since, so I don't know if that is still the recommendation. Not sure if iphones maintain connection with the watch in airplane mode or if you have to manually turn bluetooth back on, either way it should still work as all processing is done on the phone.
You might be having syncing issues because you're watch is several generations old and just doesn't have the hardware to keep up with the app, just a guess, someone else who has a watch might be able to answer that better.
@swupnil I love the app but having issues with tagging. For some reason my last session was low res (iPhone 12 Pro so idk why), and the initial tags missed some. I've tried on my iPhone, iPad Pro and MacBook within the app to 're-tag shots' but it only allows me to unselect. When I do, 1) I can't reselect the shot if hit by mistake/change my mind, 2) I can't tag the right person on the shots that are them, and 3) when selecting between players it doesn't change the shots selected (i.e., if I switch from me to the other player, the selected shots for me remain highlighted, so it compiles both selected shots under one rather than differentiating between the two players). Happy to clarify more if this is new info/isn't clear, and certainly would love to know if I'm doing something wrong or missing something. Thanks!
@swupnil disregard the last post, figured it out. I saw in another post that the mph differs from a radar gun because you’re taking average velocity versus initial velocity off the racquet face like a radar gun uses. However, is there a coefficient or some sort of calculation to figure out that initial speed to compare to what you’d get from a radar gun? I realize that the point is to compare speed over time, but on serves specifically I’d like to know what they would be clocked at using the gun.
The problem is that in order to be accurate at all you'd have to factor in wind speed + direction, and even ball spin, temperature, and humidity to be even more accurate
swupnil
Rookie
@essoclub @jasonw221 we actually did some research on this in the last few months, comparing initial speed to average speed, both calculated from high speed cameras. The relationship is surprisingly linear but there is more noise (ie variance along the regression line) at higher speeds than at lower speeds. With that said, it looks pretty damn good and we’ll add this as a conversion option in the app soon 
@elsk we haven’t gotten our hands on one yet but I would like to think Google knows what they’re doing, so yes this gives us the green light for Android! We’ll likely wait until our next fundraise (and also for a couple more flagship devices from other manufacturers) before we start development on that, so I’d estimate a launch in the second half of 2022 to be safe.
@elsk we haven’t gotten our hands on one yet but I would like to think Google knows what they’re doing, so yes this gives us the green light for Android! We’ll likely wait until our next fundraise (and also for a couple more flagship devices from other manufacturers) before we start development on that, so I’d estimate a launch in the second half of 2022 to be safe.
time_fly
Hall of Fame
Three more things I'd like to see:
First, the easy thing: the software currently reports the percent of cross-court and down-the-line forehands and backhands made, but I'd like to know the percentage attempted. Am I hitting cross court or down the line too much? Does my opponent have tendencies? A similar report on inside-out / inside-in groundstroke attempts made and hit successfully would also be cool. These are all just based on the type of shot hit, the hit position, and the bounce position so it should all be pretty easy to calculate and report.
With respect to ground stroke statistics, I'd like to be able to see return-of-serve separately from rally shots. Should be pretty easy; just need to check if the incoming shot was labeled as a serve.
Finally -- and this is a slightly bigger ask in terms of developer effort -- I'd love to be able to mark winners and unforced errors while reviewing video. It's just too much playing with the watch during points, and I find it distracting but still want the info later.
PS. Can't resist adding to the wish list now that I'm typing it: In the spirit of Craig O'Shanessy, it would be great to see more stats broken down by rally length. Who is winning the short vs. long ralles and how often do they successfully execute a "+1"?
First, the easy thing: the software currently reports the percent of cross-court and down-the-line forehands and backhands made, but I'd like to know the percentage attempted. Am I hitting cross court or down the line too much? Does my opponent have tendencies? A similar report on inside-out / inside-in groundstroke attempts made and hit successfully would also be cool. These are all just based on the type of shot hit, the hit position, and the bounce position so it should all be pretty easy to calculate and report.
With respect to ground stroke statistics, I'd like to be able to see return-of-serve separately from rally shots. Should be pretty easy; just need to check if the incoming shot was labeled as a serve.
Finally -- and this is a slightly bigger ask in terms of developer effort -- I'd love to be able to mark winners and unforced errors while reviewing video. It's just too much playing with the watch during points, and I find it distracting but still want the info later.
PS. Can't resist adding to the wish list now that I'm typing it: In the spirit of Craig O'Shanessy, it would be great to see more stats broken down by rally length. Who is winning the short vs. long ralles and how often do they successfully execute a "+1"?
rider4ever
Rookie
@swupnil any advantages to upgrade to the latest iPhone version? I am currently using the iPhone 11
swupnil
Rookie
@time_fly we're working on all of those things! There is a major redesign of the heatmaps / shot analysis in the works and it will let you drill down into tendencies on serves, returns, serve+1, return+1 and so on.... and you'll be able to visualize heat maps with respect to points won vs lost. It's going to be a complete game changer and basically give you a playbook of exactly which strategies you need to execute to beat a particular opponent. Once we release this, it will be a very severe disadvantage if you are NOT using SwingVision
@rider4ever biggest improvements will be way better battery efficiency, vastly improved ultra wide camera (if you find that you use the 0.5-0.9x zoom for your courts) and future-proofing for things like live streaming and real-time line calling (as opposed to line challenges, which are already supported on iPhone 11 + Apple Watch Series 5 / SE or newer).
@rider4ever biggest improvements will be way better battery efficiency, vastly improved ultra wide camera (if you find that you use the 0.5-0.9x zoom for your courts) and future-proofing for things like live streaming and real-time line calling (as opposed to line challenges, which are already supported on iPhone 11 + Apple Watch Series 5 / SE or newer).
happyloman
Semi-Pro
@swupnil
I have three sessions (60 fps and 1080) that are between 1.6 to 2.0 Gigabytes in size, and they have been stuck in the "Your session data is currently uploading" state for 24 hours.
Previous sessions (30 fps and 720) have not had this issue.
My internet service provider upload speed is 10 Mbps, so 24 hours would have been more than enough to upload them
Any ideas?
I have three sessions (60 fps and 1080) that are between 1.6 to 2.0 Gigabytes in size, and they have been stuck in the "Your session data is currently uploading" state for 24 hours.
Previous sessions (30 fps and 720) have not had this issue.
My internet service provider upload speed is 10 Mbps, so 24 hours would have been more than enough to upload them
Any ideas?
rider4ever
Rookie
swupnil
Rookie
@rider4ever I think they both have the same ultra wide lens? Maybe I'm forgetting. Overall I think things will be pretty similar for both with respect to our roadmap over the next year.
@happyloman there's a neat "hack" that gets the uploads going even faster:
1. Plug your phone into a charger
2. Force quit and reopen the app
3. Leave the app open
Uploads should fly! Longer term we're looking at ways to only upload the footage with the dead time removed, which should hopefully make these go around 5-6x faster!
@happyloman there's a neat "hack" that gets the uploads going even faster:
1. Plug your phone into a charger
2. Force quit and reopen the app
3. Leave the app open
Uploads should fly! Longer term we're looking at ways to only upload the footage with the dead time removed, which should hopefully make these go around 5-6x faster!
rider4ever
Rookie
@swupnil The ultra wide lens is a lot better in the iphone 13 pro f/1.8 aperture in the pro vs the f/2.4 aperture in the 13 among other improvements to the iphone 13 pro ultra wide lens. 13 basically has the same lenses as the 12 pro while the 13 pro get a new lens. I don't mind spending the extra couple hundred to future proof. Let me know if the newer lens would make a difference.
iphone 13
- Dual 12MP camera system: Wide and Ultra Wide cameras
- Wide: ƒ/1.6 aperture
- Ultra Wide: ƒ/2.4 aperture and 120° field of view
- Pro 12MP camera system: Telephoto, Wide, and Ultra Wide cameras
- Telephoto: ƒ/2.8 aperture
- Wide: ƒ/1.5 aperture
- Ultra Wide: ƒ/1.8 aperture and 120° field of view
swupnil
Rookie
Ah wow @rider4ever ! Totally missed that. Then yes definitely go for the pro. I think that will make a big difference in the tracking accuracy, line challenges, etc. if and when you need to use the ultra wide lens.
mnguyen30395
New User
Would an Android app be prioritized over a Windows app?
@essoclub @jasonw221 we actually did some research on this in the last few months, comparing initial speed to average speed, both calculated from high speed cameras. The relationship is surprisingly linear but there is more noise (ie variance along the regression line) at higher speeds than at lower speeds. With that said, it looks pretty damn good and we’ll add this as a conversion option in the app soon
When you finally roll this out, I'd appreciate a write up on your findings. There were a lot of discussions and private messages flying around on this topic and how to build a function to correctly take the average speed back to the initial. Hopefully you're tackling this issue for ground strokes as well as serves. I know it's more difficult because the variance in ground stroke distance is a lot bigger as compared to serve distance. I basically just add 2 mph per 10 feet traveled with slight scaling for speed. 1.8 for a 50 mph average speed ball and up to maybe 2.2 for a 90.
swupnil
Rookie
@mnguyen30395 yep Android will happen for sure. I’d say there’s a 1% chance we ever make a Windows app. We were never going to make a Mac app - only reason we did it is because it was like 3 lines of code to run the iOS app on Mac.
From the data we see among ios/Mac folks it’s 99% of people recording with their smartphone and 1% importing from a GoPro - real-time processing on smartphones is the future (and necessary for things like line calling, real-time feedback, etc.).
@Demented I’m not sure how much we’ll share for competitive reasons.. but we have collected initial and average speeds for thousands of serves and groundstrokes with various speeds / distances from beginner up to pros. I believe we will have the best conversion possible given our data set.
From the data we see among ios/Mac folks it’s 99% of people recording with their smartphone and 1% importing from a GoPro - real-time processing on smartphones is the future (and necessary for things like line calling, real-time feedback, etc.).
@Demented I’m not sure how much we’ll share for competitive reasons.. but we have collected initial and average speeds for thousands of serves and groundstrokes with various speeds / distances from beginner up to pros. I believe we will have the best conversion possible given our data set.
rider4ever
Rookie
How about giving an option to add scoring manually in the processed video? Like choosing who won each point while watching the playback?
Is the 13 max much better than the 12 max for this app?
antony
Hall of Fame
I've heard the magnification for in between zoom lenses has improved. 0.7x on the 12 pro max was kinda grainy and I heard that improved in the 13 pro
I'm glad that someone finally agrees with me on the initial vs average debate. If I have to watch my ugly ass strokes/footwork then I should at least get the pay off when I hit a triple digit serve or a 75+ mph forehand.
@swupnil, I've been wondering lately about my camera positioning. I've been using a top of the fence mount at about 10 feet for the last year and I can't get it any higher. Do you have accuracy numbers for the different mounting heights? Like 5 feet is 90% in the near court and 92% in the far court while 10 feet is 97% in the near court and 85% in the far?
@swupnil, I've been wondering lately about my camera positioning. I've been using a top of the fence mount at about 10 feet for the last year and I can't get it any higher. Do you have accuracy numbers for the different mounting heights? Like 5 feet is 90% in the near court and 92% in the far court while 10 feet is 97% in the near court and 85% in the far?
mattbrockman
New User
@swupnil Hey, first off, love the app. I use it all the time. Looking forward to the future update mentioned that will show top speed rather than average.
I use it on a MacBook pro (2020 M1 16GB) and run into the following issues frequently. Is there a workaround or future update that will address these?
1) Out of memory (I forget the exact wording, it is an error message that comes up in red text on the bottom while in the "on-device processing in progress" phase). It happens very often. Never the first time processing, but each time after that (without restarting the computer) with a new video it is more likely to happen. I don't know exactly how often on the 2nd or 3rd video but by 3 or 4 it rarely finishes without the error coming up and once the error comes up once it never works after that until a restart. If restart swing vision (without restarting the computer) and try again as the error message suggests but that never works. The only solution is to restart the computer and that always works (but only for the next 1 or 2 or 3 videos before it happens again). I don't have a lot else going on at the same time other than a bunch of chrome tabs (I've tried closing everything but swing vision and I still get the same memory error and the only solution is to restart the computer).
2) The accuracy of each individual shot is not very good. If I go to the edit mode it is almost always a little off (and sometimes way off) on all parameters, hit timing, bounce timing, court location, and sometimes (but less often as the earlier parameters) shot type. I can't trust it for speed or line calls for any particular shot. It doesn't appear to be as accurate as some percentages I have seen earlier in the thread. It is great for getting a general sense of the match and comparison to my opponent or my previous matches. But it would obviously be great if you could trust each individual shot without going to edit it and fix all the issues. Not sure if this is a MacBook app issue or a gopro issue or a general app issue? I don't have an iPhone hence the gopro/macbook setup. On the gopro I use the following settings: 1080, 60FPS, 1.0 zoom. I mount it on a QM-1 fence mount which with the standard tennis fence doesn't get it high enough to get the line of sight suggested in the app but there is not really anything I can do to get it higher. I have played on courts a couple times with higher fences but I don't remember there being a big improvement in accuracy.
I use it on a MacBook pro (2020 M1 16GB) and run into the following issues frequently. Is there a workaround or future update that will address these?
1) Out of memory (I forget the exact wording, it is an error message that comes up in red text on the bottom while in the "on-device processing in progress" phase). It happens very often. Never the first time processing, but each time after that (without restarting the computer) with a new video it is more likely to happen. I don't know exactly how often on the 2nd or 3rd video but by 3 or 4 it rarely finishes without the error coming up and once the error comes up once it never works after that until a restart. If restart swing vision (without restarting the computer) and try again as the error message suggests but that never works. The only solution is to restart the computer and that always works (but only for the next 1 or 2 or 3 videos before it happens again). I don't have a lot else going on at the same time other than a bunch of chrome tabs (I've tried closing everything but swing vision and I still get the same memory error and the only solution is to restart the computer).
2) The accuracy of each individual shot is not very good. If I go to the edit mode it is almost always a little off (and sometimes way off) on all parameters, hit timing, bounce timing, court location, and sometimes (but less often as the earlier parameters) shot type. I can't trust it for speed or line calls for any particular shot. It doesn't appear to be as accurate as some percentages I have seen earlier in the thread. It is great for getting a general sense of the match and comparison to my opponent or my previous matches. But it would obviously be great if you could trust each individual shot without going to edit it and fix all the issues. Not sure if this is a MacBook app issue or a gopro issue or a general app issue? I don't have an iPhone hence the gopro/macbook setup. On the gopro I use the following settings: 1080, 60FPS, 1.0 zoom. I mount it on a QM-1 fence mount which with the standard tennis fence doesn't get it high enough to get the line of sight suggested in the app but there is not really anything I can do to get it higher. I have played on courts a couple times with higher fences but I don't remember there being a big improvement in accuracy.
As you can tell from my previous post, chasing accuracy is part of the draw backs of a single camera system. I've arrived at the conclusion that 10 feet is just not enough height for the system to properly call the far court bounces. The net top occludes a wide section of the court between the service line and the base line@10 feet. My thinking is that 12 feet is really the minimum or you're better off aligning the camera to see under the net and capture the entire court. This then introduces a lot more occlusion from the player blocking the LOS of the camera.
Similar threads
