Assuming you will continue to improve your strokes and with high consistency, the best way to get to a higher level is to play against opponents at a higher level. If you are a 4.0 and want to be a solid 4.5 then only play against 4.5s and 5.0s. Yes, you will lose many more times than you will win, but in a tournament against a 4.0 you will have no problems.
I think advice about playing higher level of competition is basically good. But, do not agree that you should play only against higher ranked players. A lot has been written about knowing how to win. You need to play players of even level to know how to play and win in a close tense match. You also need to play players at a little lower level than your level to know how to handle the pressure of expectations - you are expected to win. Many players choke if a player perceived as "lower level" starts playing them close.
If you are practicing with a player at a lower level, play their strengths such as hit everything to their forehand to get a good workout. Or, work on your weaknesses such as playing serve and volley.
So, yes, play up a fair amount of the time, but play at your level and below your level too.
4.5 is just the "best of the hackers"
5.0 is where the real tennis starts, preferably 5.5.
*sips whiskey*
I think you discovered a few things about practicing. The type of practice you do affects the results you get. Also, it takes a while of consistent practice to see results.
Even though you didn't get the results you were hoping for, you should still be happy one aspect of your game improved so much. I would not completely stop what you were doing since you don't want your gains to disappear and go back to your baseline level. Instead, just go into maintenance mode meaning do it less often like maybe once a week, and don't try to keep pushing yourself in that area.
So the question is, "what should I be practicing/drilling to improve?" Actually, you first need to ask yourself, "what part of my game should I improve to get to the next level?" I will say that I don't recommend weight lifting to improve the pace on your groundstrokes or serve. That is just so naive, and hopefully with the internet, that advice will go the way of the dodo so people don't waste their time.
Since your goal is to get to the next level (another question people need to first ask themselves: what are their goals) and based on what you said, I would recommend improving your serve first. Even that is a giant task that you will need to break into subtasks. For example, you need to develop a better second serve and work on spin. Then you need to develop your first serve as a weapon by adding pace. Then you need to be able to place your first serve. After doing all that, then you can focus on the other parts of your game like your return of serve. Or you can decide to improve your serve more!
You'll have to set some serve goals like I want to serve 90mph or practice my serve for 3 months. Your serve regimen could be practicing 3x a week for 20-30 minutes each time. Practice ad+duece side hitting cones placed on the T, middle, and corner. Maybe practice topspin serves until you can get 90% of them in.
You don't have to follow my advice; This is a rough plan anyways. But hopefully this will give you an idea of what you have to do to improve. With a good plan, I believe most people will improve faster than just mindlessly hitting and practicing a lot. Who knows? Maybe you'll decide to use your newfound footspeed and be a counterpuncher or play defense like Nadal!
Wow a bunch of great replies already! I think I should provide more background information though, after reading all these.
My groundstrokes feel pretty good. Last summer I worked intensively with a college coach to practically rebuild them, and they have improved immensely since then. My forehand sometimes breaks down though, especially when I get soft balls that I should put away or at least set up put aways, and also it breaks down after a few hours of playing; it gets all spinny with no depth. My backhand is just recently becoming a weapon. I don't hit too many errors with it, but it's definitely not easy to work a point with it.
My volleys are pretty good, nothing amazing but I don't botch too many. I just struggle with really taking advantage of them; I think I just need to take the risks, but all too often I put them just a little too close and my opponent can get them back sometimes. My overhead is really pretty good, I feel like if I get an overhead the point will almost always be over. I do have trouble with them when I have to travel backwards and lean backwards to hit them, but as I said I'm not overly concerned about my overhead.
Aaaah my serve. So off and on, this is definitely what I need to work on the most. Yesterday I played a proset, and I had at least 15 double faults. Now that's not normal at all for me, but I'm giving you the worst example so you can see just how bad it can be. (I don't want to fluff up this story, I'm really here to improve and get better!) My first serve I can clock in at around 80 mph usually, I have a solid kick serve, I'm working on a twist, my slice is pretty weak, and my flat is pretty inconsistent. My second serve really sucks, thus the double faults. I try to hit a mix between twist and kick, but it goes long whenever I do double fault. I really REALLY want to develop my serve into more of a weapon and at least something that I can always rely on. I'm not there at all yet.
My return of serve unfortunately isn't very good either. I've really been trying to work on this and hit some semi aggressive returns. Before I would usually slice almost all of my backhand returns, not even very deep. My forehand would have topspin but again rarely be deep or low. I'm trying to get rid of the slice unless I absolutely need it, and trying to get both wings to return deeper and lower.
My endurance, it's pretty good I suppose. I'm 5'11" and 165 lbs. I can play a two hour match and still move quite well, though my shot placement and depth starts to suffer.