With all due respect to Fed, that he would have 4-5 FOs in this era is as much of a reflection of the CC field as it is of his ability on clay and his famous (and remarkable) conistency. Not to mention that I don't subcribe to that reasoning in general, following the same logic we could also put Murray above Lendl and Agassi and shoulder to shoulder with Borg, it falls in the same basket as Nadal's 30 moral slams as far as I'm concerned. If Fed had pushed Nadal to several 5 setters I could maybe see it but that he didn't is not simply a consequnce of his mental failures against Nadal but also due to him not being a natural CC player (yes, I know he grew up on the surface).
Yes he did beat Novak in 2011, by banging down 20 aces in faster conditions. Mind you, what I said for Fed goes for Novak too (maybe to a lesser degree depending on whom you ask), there's a number of reasons of why he lost 2015 FO final to Stan but one of those not often mentioned is that the latter just hits a much heavier ball which is rewarded on clay. Because Novak is like Fed, they play HC tennis on clay and get away with it because of their overall ability and lacking depth of the field on clay. Compare Novak's match with Stan the year before in WTF indoors when Novak bagelled him, it's a world of difference.
As
@Hitman already noted, Fed was one of the main favourites for FO in 2004 (especially after beating Coria in Hamburg final, a BO5 match) and was absolutely decimating the tour, it's one of his career most dominating years. That he went down to (way past his best) Guga at FO in straights is telling.