You have a few options here:
A) Do as you did. Allow your opponent to make the call and that's that.
B) If you returned the ball in the court and your opponent didn't play it, take the point. This will involve some discussion, possibly an angry exchange, you might even have to pull out a rule book, but according to the rules it's your point and you have every right to take it.
C) Allow your opponent a second serve just this one time after explaining the rule to him. Again, this might require a review of the rule book, but if your opponent protests too much you can decide to revert to option B.
I find that most of the gentleman I play against are really good people and so I'll usually go with option C. Every once in awhile I come across someone who rubs me the wrong way and in those cases I'll go with option B from the get go. Only in non-league social matches will I select option A.
Read my blog for club and USTA players at CourtCoverage.com