Raul- both teams call things that happen on their own side of the net because they are in better position to see that.
On a let call there is no "side of the net" and instead of relying on visual evidence it is relying on audio evidence. Better to let anyone call it to have a better chance of it being called correctly. I simply can't see how the serving team would ever gain an advantage by calling a false let (as long is it is called in a timely manner).
So lets say that you remove the ability for the serving team to call a let. There is a serve that basically just slows the serve down but the receiving team wasn't sure and gave the benefit of the doubt and let the play proceed. The receiver cracks a winner. Do you really think that wouldn't cause a huge argument?