Djokovic, Nadal and Murray are without a doubt the toughest matchups for Federer.
Berdych has won two in a row against Federer. I wouldn't say Federer looked particularly sharp in either match and still nearly won the recent one in Dubai.
After that, I'd group Berdych, Delpo and Tsonga together because all are capable of overpowering Federer if they get hot.
After that, there's no one worth listing, IMO. Certainly others can and have beat him, but it generally takes a superb effort from them and/or Federer being off his game.