Yeah this is where their careers differs the most imo.While we are on the subject of age.
Djoker won 12% of his slams while Fed was in his 20s.
Fed won 40% of his slams while Djoker was in his 20s.
You can make a point that Federers closest ATGs are 10/11 years older than him (Pete/Agassi), and therefore he had an easier breakthrough than for example Novak.
But then you must also see that Federer already in 2006 was being chased and denied slams by a younger hungry ATG, and later by another one.
I always find it interesting when people claim Federer had it easiest, when he was denied slams almost every year from 2006 and on, by two younger players which turned out to be two of the greatest players in history.