You are wrong- a let can be called by anyone on the court. I'll routinely call a let for balls that roll through the back of the opponents court- if I am sitting there trying to decide whether the ball could potentially injure my opponent then I am clearly distracted by it and its far better to just call the let and play the point over.
from the code:
Lets called when balls roll on the court. When a ball from an adjacent
court enters the playing area, any player shall call a let as soon as the player
becomes aware of the ball. The player loses the right to call a let if the player
unreasonably delays in making the call.