I think you really should change your ritual. At least raise your hand or racket until you are truly ready, like the others have said. If you want some extra time to gather yourself then pace around, turn your back to the server or fiddle with your strings. If your body is facing the server, you are shuffling your feet or rocking back and forth, and you have both hands on the racket whether or not you are looking up down or to he side to most people you look ready to receive.
Take a look at it from the other guy's perspective. Maybe the guy wasn't trying to quick serve you. So he sees your body looking like you're ready, he hits good first serve and then you claim you weren't ready. What? OK, not a big deal. Then it happens again. Now maybe he thinks you are the one who is trying to pull a fast one out there.