Well i hope i was playing with players who are better than me and have some passion to improve their game. My hitting partner is one of my oldest friends and he just consumes my energy on the court. He doesn't want to improve, he still arms the ball after numerous advices, i feel like i'm forcing him to play. It is really hard to improve yourself if you have a hitting partner like that.
The improvement comes with the challenge. Not only with tennis, it's my general rule in life. You can only improve when you need to improve. So i'm pretty sure you will gain a lot more by playing better guys than you. The problem is if they wanna play with you or not. If they are your bros, than even if they don't, continue playing with them. But if they are just strangers, than you should watch for their reactions. To be the unwanted man can threat your comfort level and it's just a better idea to find somebody else to play. If that is not the case, go on, i'm sure you will improve a lot if you have the passion for it.
Also i'd suggest going for singles more. If you suck at net game, either take some lessons or just go for singles and be the defensive player that don't approach net so much. You have the physical advantage, so maybe you should consider making your playing strategy based on your power. You can change to a crazy topspin game and send them your bombers, just wait for mistakes. This will also give you some more consistency. You say you have a good serve. You are tall too and you don't have an excuse to make it a true weapon that everyone is scared of because you can simply practice it by yourself.
A man can't do everything well. You just need to know your own weaknesses and hide them with other aspects that you do perfect.