Djokovic redirects the ball better than anyone in mens tennis. Its the reason why hes about the only guy to get Rafa to run out of gas during a match.
Wind gets in the way of that.
Also traditionally (outside of God-mode 2011 Djoker) he has been a guy who can go through patches of lots of UE's and that also worsens during wind.
Andy Roddick -- "Im one of the lucky few that gets cheered for" -- after loosing to Roger Federer