I'm going to say clay because it requires so much patience and you can't end points as quick as on hard or grass courts. Personally, I grew up on hard court and my game drops when playing on clay.
I'm a Fed fan but don't forget Fed grew up on clay in Switzerland; made 4 FO finals and won one. If it wasn't for Nadal, Federer would arguably have 3 or 4 FO trophies.
Clay is no doubt the toughest & most challenging surface for the all-time greats - Fed, Djokovic, Sampras, Agassi, McEnroe.
Nadal still makes grass and hard court slam finals (and has won) at a lesser clip than his clay record. However. it shows you he's not one-dimensional like Guga or Burguera were and that a person is able to cross-over with successful results on all surfaces.