You go at a servers *reasonable* pace. If you are wandering around the back of the court delaying, you are the problem. If you are walking to ready position, and they're rushing you, they are the problem. If you face someone who rushes you, hold your hand up or simply do not face them until you are ready. If they start the motion before you are set (i.e. they start right as you turn around), hold up you hand. You might want to point out that you don't like being rushed as well, that might slow them down if they aren't doing it on purpose (and maybe if they are!).
Personally, if I get someone I sense is doing it on purpose (or stupid enough to do it accidentally), I have no mercy. I'll hold my hand up at the last moment, ensuring they waste energy with a serve. I'll start meandering around the back of the court, slowing them down even more. I would never do this to a reasonable server, but people who rush me get me absolutely furious. It's incredibly rude.
2 x [Babolat Pure Storm Limited] , MSV Focus Hex 1.23 @ TBD