Peak Hewitt, Roddick, some appearances from Davydenko, Nalbandian and Gonzalez, and Safin. Federer's competition is at least equal or above Murray. Nadal was around then too but was still a baby, he had the game to trouble on hardcourts by 2007 though but he lost to Gonzalez at the AO. Beat Federer almost every time they met though.
Hewitt had the potential to have a great run at the AO in 2004, but Federer was pretty much in Godmode during their match and beat Hewitt easily after he took the first set. Hewitt reformed and came into the AO 2005 ranked and seeded much higher than the year previous. This paid off, as he avoided Federer and fought tooth and nail against prime Roddick in the semifinals, winning in 4 sets. This set up a final against Marat Safin.
Hewitt fell off after AO 2006, but he still made some deep runs after his decline. Made the QFs of 2006 Wimbledon and the 2006 US Open. **** poor performance in master tournaments though. Fell off for a good period after 2006-2008, had one last good run at Wimbledon in 2009 and it looks like he is pretty much done now.
Safin is an interesting case. Had a wonderful run to the final in 2004, but lacked the fitness to take the title away from Federer. Federer also played outstanding tennis and made it very difficult for Safin to get into the match. Safin came back in 2005, seeded again after being ranked 86 in the world in the '04 final, he came into the semifinals of '05 AO seeded 4th and ranked 4th in the world and totally on his game. He just managed to take Federer out in a very, very tight 5th set.
Roddick was on his game from 2004-2007, those were probably his best years on tour. After 2010 though, Roddick clearly lost his edge and was in no way a contender for the slams. Roddick never had a chance at a slam during the Federer era as his game did not match up well against Federer and he declined faster than and his talent was much less than Federer's.
Davydenko had 5 great years from 2005-2010, made it deep in several tournaments/slams. Never made a slam final though. Did not have the mental game to take Federer out whenever he met him in a slam.
Nalbandian had a great run at the AO in 2006, but he was a mental midget and lost to Baghdatis. Could have been his next shot at a major.
Murray played great up until the finals of the 2011 AO but lost miserably to Djokovic. Equal to say, Federer against Davydenko in any slam.
Djokovic lost to past his prime Federer at the 2011 French Open. Made it up at the 2011 Wimbledon Championships by taking Nadal out in the final, though. Nadal was in his second US Open final and lost this one to Djokovic over a hardly contested final, and that is something Federer did lack, competition that could push him in slam finals.
Federer had more variety, but Djokovic did have more competition.