Sorry, your interpretation is just not correct. Saying "I didn't see it" means you have no opinion, and thus there is no potential for disagreement.
Just think through my scenario (player A at net looking forward while partner B makes the out call at the baseline), if your interpretation were correct:
Opponents look at player A and ask what he saw - knowing of course that he didn't see it since he was looking forwards.
Player A says "it's my partner's call" or some such, to avoid the "trap" you're cautioning against.
Opponents ask player A point-blank "did you even see the ball land? Yes or no?"
Now, according to you, he can't say "no, I was looking forwards at the time" without conceding the point. And he can't say "yes" (without lying).
Of course that's ridiculous. And while not every USTA rule makes sense, they are not *this* ridiculous
The rule that it's sufficient for one partner to make the call in dubs is acknowledgement that in dubs it is legitimate and fairly common to have one partner looking ahead and not watching the ball land behind them.