You definitely need to get fitter, since you've chosen this style of play. Being a pusher, fitness is the main attribute. When you see that other guy falling on his face after 20 exchange shots or so, you should be barely gasping for air and still manage to have relaxed grin on your face. That puts the other guy even further down into the hole mentally and physically.
Ok, seriously, having this style drives a lot of players nuts but you also have to be prepared, it will lose its effectivness (if you find boring at winning) and decide to try higher levels. These guys don't buy any of your shots. You'll get wipped off the court really quick. Pushing has its limitations in general. What I mean, eventually you will have to pick up the pace and accuracy, which usually screws up consistency, pushers are known for. This is, why most players try to learn hitting the ball the way pros do so they can dictate points. If you end up playing one of these hard hitting but consistent guys, you will have no chance dictating point.