That's the real question, and therein probably lies the way to think about the answer. Why is there any rule about a let at all? Why not just give two serves only, regardless of any delay?
My opinion is that it relates to muscle memory. You take a first serve, and if you miss it, you have an idea about how to adjust your swing, feet, ball toss, etc. for the next serve. If you get interrupted or delayed (through no fault of your own or your partner's), you have lost that muscle memory, and your chances of double faulting are worse. If the server is the person who has to move his feet to clear an outside ball, no matter how quick it is, I always give two serves, because I think the rules and the Code expect a server to get one shot, then a legitimate chance to make adjustments before the second serve.
This same idea applies to the server's partner. His job is to clear first missed serves quickly, hold the extra ball, and do everything he can to let the server concentrate and think only about his serves.
It's the same as throwing darts. You usually throw three in a row, making tiny adjustments to your second and third tosses as you go. If you had to move around and take delays in between tosses, your darts score would be much worse.
We're pretty generous at our club about giving first serves. I have never asked for a first serve, however, and I have sometimes declined it when the other team offered it due to delay caused by me or my partner.
One time while playing another team, the other team's captain came through the gate on our court while my partner was between serves. The capt. took his sweet time re-latching the gate and walking off our court. My partner turned to the other team and said, "Where are we, fellas?" They said, "Thirty all, second serve." My partner said, "No class", then served.