The least amount of points that someone has to get in order to win a set is 24. If he's being a pusher, then he's not hitting winners (unless you're at the net and he hits a defensive lob and you can't get to it). This means that his points are the result of your errors.
Are you saying that he just keeps the ball in play long enough for you to make a mistake?
Obviously his strength therefore is to keep the ball in play, which is not your strength. Your strength is less a rally with less than 3 shots. So you have to figure out how to make most of the points end in less than 3 shots.
This will not be done by playing a baseline game. You're going to have to incorporate more drop shots, more slices, spin, and other tactics to get you to the net and put away the shot easily.
If you hang back at the baseline the whole time, then the only way you'll beat him is if you become more consistent than he is.