When I see this sort of thing from opponents, I'll tell them to just leave the ball if it's way off the court against the fence. On my "good serving days" it is not at all uncommon for me to encounter opponents who will employ all manner of gamesmanship to try to knock me off my rhythm - including running two courts away to get a rolling missed first serve that went wide. On such occasions I usually ask them to just leave it and get it later, if they persist, I'll ask them for a first serve, if they refuse, I guarantee their next couple serves are going to be "out". I only had to resort to that once in a match a couple years ago... after that the nonsense stopped.
What REALLY cheeses me off though is when I'm serving in dubs and my own partner is doing it all wrong... either chasing every ball, no matter how far away it is, or not chasing any of them and leaving me, as the server to gather every netted first serve.
When my partner is serving in dubs, and they hit their first serve into the net, maybe it bounces and rolls slowly back up the middle of the court, not exactly next to ME, but it will take a bit for it to get to my serving partner... or maybe it's bouncing around short and wide into the service box on the other side of where I am... I make the effort to go and get these balls, unless they bounce so quickly past me that they'll go well off court, or right back to my serving partner long before there would be any sort of delay in them getting their second serve ready. If the serve goes into the net and bounces short in front of me, I either brush it off the court to the side, or hustle and pick it up - no way do I want that ball directly in front of me to be any sort of distraction to a potential poach or net play for me. All in all, my objective is to do what I can (more than "my share" if needed) to allow my serving partner to have a serving experience that is as fluid and uninterrupted as possible.
When I am serving, I do expect this from my partner. When I play mens I find about half of my partners "get it" and about 10% or fewer of my mixed partners... the only one I know that does "get it" is my wife, becuase I've told her about what I do and what my expectations are. Once in a great while, a mixed partner will go chasing a ball I've served that hits the net and goes wide into the neighboring court or way off court, and whenever I mention that they don't need to chase that one, they're holding me up, they then stop touching any netted first serves even if the ball is rolling right in front of them.
I don't know why this whole thing is so difficult for people...