In a league match, my partner serves and as the ball is being returned to him I turn around and see a ball rolling on our court. Thinking it came from the adjacent court I call a let. However, our opponents say the ball had fallen out of the servers pocket. They claim the point. One more thing. Apparently, after the ball comes out of the servers pocket, he momentarily stops play but then changes his mind and hits the ball back. The opponent stops and does not make a play. In the ensuing discussion the server says the return was out and the receiver says the return was clearly in. I saw it in. I said play a let but the receiver claims the point because the server looked like he stopped play. I relented and gave them the point. Was a let the proper call?