Check your toss. A lot of rec players toss to a certain position in front of the server in the court, which works for the deuce court (serving from the right side), but is way out of line for the ad court (serving from the left side). Try letting the ball drop to the ground to see where it falls relative to your body.
Per comments above, your stance orientation may also be slightly off. For me, as a righty, my shoulders are generally pointed down the T on the deuce side and at the right net post on ad side. I see lot of people who set up wrong on one side and, consequently, really struggle on that side.