The players who could unsettle Djokovic:
1. Nadal on clay and on a good day on the other surfaces too. On clay he is nearly unbeatable when playing well, no matter what Djokovic or anyone for that matter does. On the other surfaces, on a very good day, Nadal can beat Djokovic through physicality and dominating with his FH, especially DTL. Ona day when Nadal is feeling the DTL FH, Novak is in big trouble. It was a miracle he escaped at Wimb this year.
2. Pre-2013 Federer. In other words a Federer who is not too old to compete with Novak in BO5. He can use his slice and variety to unsettle Novak and also dominate with his FH. Like in Nadal's case, if Fed's FH is firing, Novak is in big trouble.
3. Wawrinka. If Wawrinka is feeling it, he can hit Djokovic off the court with his heavy shots off both wings. If Djokovic is in his defensive mode, he will get exposed by Stan's power. There is not a particular wing that Novak could try to neutralize because Stan hits big and deep off both wings. Stan is the one opponent Novak can't afford to play defensively against.
Just my 2 cents. Only these 3 guys can beat a well-playing Novak IMO.