As my local Pro always asks whenever someone declares they could've beaten that pusher, "Did you win?"
If the answer is no, then you have work to do. It's as simple as that.
But, here's the weird part. You may play a 3.5 pusher and lose, but play a 4.0 topspin hitter and win. At my local club, there are many pushers. I used to hate playing them with a passion. I just knew I couldn't outlast them and that it was going to be a long grueling experience. However, I learned to relax, not overhit, move my feet, not overhit, and come to the net. I believe pushers deserve respect as tennis players, although I would never choose to hit like them.
My buddy Bob is a 4.5 (not my other buddy Bob) and he is a severe pusher. He slices his forehand and his backhand. But, he tries to get to his forehand side as much as possible. He's absolutely mastered his forehand slice and dominates 4.0's and many 4.5's. However, in practice he works on his forehand topspin shots and they are horrible. He's late on everyone of them--big time. I have played this guy a lot, so I know how to handle his shots. He beats me most of the time, but I get a win in there every now and then. Anyway, I don't really know where this message was heading, but I just wanted to express the view that if you're losing to a pusher, that means you have work to do--myself included.