Interesting question. Players can only call a let when something interferes with play on their court. In doubles, does that means the receiving team/side on a let serve? Or is the entire court the venue? If only the Server hears a let, there's an issue of his partner who is closer to the net not hearing a let occurring? And your partner did not hear a let either? That to me is rather fishy. I may let the server call a phantom let service once in doubles. But it really is the call of his partner and that is what I would state emphatically.
If you thought this was gamesmanship, you should disallow the let call since only the server heard it. His partner, who should have heard and called it, did not say 'Let!' so no let call. TBH, I played matches where one player said 'let' and the other 3 players said 'what let?' on serve.