What I mean is if you're Djokovic in 2014-2016, you're basically worried about three other players in grand slam contention: Federer, Murray, and Wawrinka. Nadal for small portions of those years (he basically went AWOL at the slam level after RG 2014). If you're Federer in 2004-2006, you're worried (at various times) about Nadal, Agassi, Hewitt, Roddick, Safin, Ferrero - all players who won at least one slam either in those years or shortly before, all holding number one either shortly before that period or shortly after. Those are the top contenders - not as consistent as the top contenders in Djoko's era, and maybe not quite at the same high level (though Nadal and Safin are at least as tough as Wawrinka), but there are more of them to run into, so the field is potentially stacked.
And Ferrer and Berdych, as consistent as they were, could never do what Nalbandian did in the fall of 2007. When he was fit and motivated, he was a much tougher match for the top players than either of those guys.