In my experience, there is a lot of variation in serve speed up to the 5.0 level. There are definitely some 4.0 players who serve harder than some 5.0 players. There are probably even some 4.0 players who have better serves than some 5.0 players. However, I rarely find people below the 4.0 level who hit consistently above 90mph (though I'm sure there are some).
Speed alone isn't what matters, but it does say something about your technique and potential ability level if you can easily bring over 100mph with good form. Perhaps all the top 50 players in the world can bring 130mph+ when they want. Federer can hit 130+ but rarely does, and he has a very strong serve.
I'd be more concerned about your second serve speed. Unless you are getting good spin, a 55mph second serve is going to be targeted by most 4.0 or higher players.
I couldn't say it any better than above. I watch our 5.0's play doubles all the time at our club. None of them server over 100mph. This might be an exception to the rule, but what I've seen. Their second serves might be 10% slower than first serves though.