Played many a pusher, and this is probably what your not doing that seems mystical to you. The main target area they have is the very center of the T. If they aim there they can never lose a match!(its very far in any direction that they will even get CLOSE to a line from there!!)
I have seen topspin pushers, low backspin pushers.. but, the one thing they all do, is hit this middle T area. Why?.. well, first, they have cut like 8 to 10 feet from you. So, those powerful baseline to baseline groundstrokes you have been working so hard on with your hitting partner, now U simply can't hit them because of the lack of physical distance from the T to the baseline. The better pushers hit low, slow, backspin JUNK to that spot, so now U have to hit up to them, or, if u try to blast to a corner, if not used to playing pushers and adjusting, your ball, will go a foot out, and a foot to the right(because with no weight to the ball, your follow through tends to do this).
They are a maddening lot, and enjoy taking a previously thought "superior player" right down to this ugly, looking nothing like pro tennis, match. That is how they win. They take a tough guy with tons of skill and talent, and soon enough, he is blocking and bunting his stroke just hoping to keep the ball in and hoping this madness is OVER while Fbombing like a madman!(and looking like a LESS SKILLED PUSHER HIMSELF, which is also humiliating!) haha (Ive been there myself, many a times, so i talk from EXPERIENCE!)
I will finish with, I would rather play pro tennis, if I could. Why?.. Not many pushers there, everybody goes for the same type of aggressive play, and so you can see consistently the same type of ball, on most of the days(until, say, Rafa, who wins because its tough to prepare for that much more topspin than a regular shot, and so most pros shots are just off enough to go out)
Wait, one more, now everyone will tell me I have to develop an awesome approach shot and volley/overhead to beat them.. developed that, got it. I just wanted to clarify as I have analyzed these fools(who aren't that foolish, as they don't have to be as athletic or strong to pull off this "loophole" in an otherwise fantastically FUN game to play!)
So, yeah, next time try this simple, painful strategy they all employ and you too will be an annoying little guy I would rather beat up than ruin the game of tennis by having to play! Good for you! haha