chris fowler and patrick mcenroe were having a discussion about a similar topic. it can be applied to OP's question i think.
mcenroe brushed on guys like tsonga who sometimes have good serving days but also hit big and play aggressively can give djokovic trouble since djokovic's return becomes neutralized.
both murray and djokovic have exceptional returns, however murray handles big servers better than djokovic and can be seen during their matches agains federer at Wimby last year.
naturally this isn't much of a weakness, since djokovic is pretty versatile from the ground, but players who serve big and can back it up are given more of a chance.
click the replay tab, followed by the 9:00 PM coverage on 1/15
the discussion starts at the 46:50 mark.