Federer still had a period of tennis where there wasn't super ATG at their peaks roaming around. He came when Sampras was ending his time at the top and Agassi was really the only one with at the tail end of his. If he didn't do anything during that time, that is on him, and this comes to the saying that everyone peaks differently. Regardless, he had several years where the players were not levels above, the way Djokovic had to deal with players being levels above him when he first came. And as I said, Djokovic had to deal with prime Roddick, Safin, in form Hewitt was still around, plus Nalbandian, then his bunch including Tsonga, Berdych, Murray etc. Novak had the toughest introduction into the tennis world than the other two ATG IMO, and it comes from the fact he was indeed younger as you said. And it is credit to Novak that he started to become a force so quickly, meaning at such a young age, he was already beating peak forms of Federer and Nadal.
Now, while I do agree that the challengers are few and far between, it is still fair to say that they all benefited at different times.