No, no no no, no. The opponents do NOT have a complaint here. What if the net man would have put away his easy sitter? Would they then have insisted the serve was out? No, they would have taken the point. If they thought the serve was out, then he had no business trying to put away that return. This is the "two chance" option, and it is what underlies about 90% of the rulings in The Code.
And you did the correct thing by not calling it out after seeing that the return looked like trouble. That would have been a late call, you had to let it play at that point. But what if the serve really was out? So what? Shots are what they're called, not what they are. That's a hard concept for some people to get, (like your opponents here). You can't have rallies and then after it's over, have someone say, "You know, I think the 3rd shot was out." If it hadn't been called out the point is on, it won't get called back like in football. If the netman thought the serve was out, he shouldn't have tried for a winner. He took his chance, he doesn't get another chance to win the same point.