Exactly! There's one match with Federer at the US Open where he breaks Federer while hardly moving from the centre of the court, because he can just put the ball anywhere from where he's standing.
I remember watching the 2004 US Open. Nobody was really expecting anything from him because everyone knew about his injuries. He had skipped Wimbledon in 2004 because of his back. The guy could barely walk.
I remember that match vividly. Even though Federer won that match in the 5th set... Federer had absolutely no answer for Agassi's ground game and return game. None. This is *prime* Federer against an injured, barely mobile Agassi returning after a layoff. The only reason Federer won that match was basically on points gifted to him by Agassi because he refused to run to get most balls and would not run to get a dropshot.
It was because of that match that I have always held that Agassi would dominate a H2H against Federer -- particular on fast/medium-fast surfaces. Federer would own Agassi on clay, though.
Another interesting thing. The only aggressive player to give Federer a real challenge in his prime was Nalbandian. Their H2H is 11-8 in favor of Federer. Nalbandian has LOTS of similarity to Agassi. Agassi was far better than Nalbandian, though. Since then, the only player to give Federer trouble (outside of Nadal) is Djokovic. Again... lots of similarities to Agassi... but Agassi a better ball striker and returner.
At that elite level, it is all about matchup. And the only guys who gave Agassi trouble were high-level serve-and-volleyers.