Without question, move up to the next level. I've played a 4.5 over the last year who was kind enough to play me when I could barely hit the ball. We're talking 1-6, 2-6, 1-6 at best. (Came back to tennis 18 months ago after 24 year layoff).
Got my butt kicked plenty, but now play at the same level with him, and have even managed to edge him the last few times. It sounds like you have a lot of natural talent, so you'll probably catch the tougher players fairly quick. It's hard on the ego at first, but it will pay off big time down the road...
(4) Yonex VCORE Tour 97 310. PPPP 18 @54/PP SG @57. Ultimate Player's Tweener.