Pushers tend to be annoying given two things:
-they force you to cover a wider area;
-they force you to adjust more precisely to a greater variety of balls.
The issue is that, often, the response is to try and hit big enough to annoy them. However, hitting hard from mid-court is something that is peculiarly difficult to do, especially when facing various spins, height and placement. On the other hand, even a well struck ball from this area do little to no damage when the opponent is in position to cover the court. Big hitters tend to play linear tennis quite a bit, neglecting the usefulness of angles; a better player, on the other hand, would play on the geometry of the court, using his better vertical position to open up the court with angles. When playing too linearly, trying to hit through your opponent, you force them to cover little to no ground horizontally and tend to hit rather deep, making vertical coverage from their part almost null. Of course, if you have to cover a few more steps forward as well as from side to side and your opponent barely has to move, don't wonder why pushers are the nemesis of ball bashers. A more cunning opponent would rather trade a little more pace for spin, play more conservatively until he gets the right ball to attack and would force his opponent much wider before finishing at the net, with an overhead or else with an easy ground stroke in the opening that will inexorably be left behind.
Tennis, when you get to hit the ball properly, is a war of position. If you get to hit the tennis ball while your opponent isn't splitting right at the center of your possible angles of replies, you've basically won the war. You can use footwork and early contact to get there; you can use angles or power; or maybe mixing the game with a few shorter slices like Federer so often did... but the whole point is to catch them out of position.
Next time you face a pusher, don't fall into playing his game. Play your game. Just make sure that you use your strokes cunningly and wait for the right ball before going for it. Most of the time, when you succeed, you realize they are no better than other players at court coverage and recovery; they just benefit from the poor tactics their opponents use to win the match. So, be smart, no powerful and work hard with your feet to always be in as good a hitting position as you could be and everything should be fine without you having to try to outpush a pusher which, more than half the time, means loosing the match.