Serving to a person I do not know (doubles). On the 4th point of the very first game the serve lands close to the line and the returner exclaims loudly, "OOOOOOUUUUU" while hitting the ball. The ball comes floating back and hearing what I thought was a clear declarative "out" do not move and wait till it lands barely hitting the outside sideline 10 feet away. There is a pause and then I say that I heard an out call. The returner says he didn't say out that he was only grunting. Now this was not a grunt. It was more like a yell. Anyway, I suggest we play a let and the server refuses and says its his point. I try to reason with him saying I had an honest belief that he said out. I asked him if he would take my word for it and he said no. I then said that the return was out and it was my point. His partner said he heard a grunt and thought the ball was out. My partner said nothing. Now at this point there is an impass. What was the fairest way to handle the situation?