There is nothing "hard" about being better with less weeks as #1. Lendl has more weeks as #1 than Nadal yet no serious person considers him better than Nadal.
The number of Slams has always been the main GOAT criterion. Other criteria like H2H in Slams, H2H overall, weeks as #1, Year End #1, Olympics, ATP finals, Masters 1000, Davis Cup, etc. are only tie-breakers in case two players are tied in Slams.
Also, the fact that Nadal has 5 Year End #1 as Federer partially compensates the difference in # of weeks. Year End #1 is equally relevant than weeks as #1, and Nadal has as many as Federer.