He does speak English quite fluently. It's just heavily accented but for someone who just learned it in his teens, he speaks it quite well.
As for the rest of the big three, English IS Fed's mother tongue. His mother is from South Africa and it is the first language he learned to speak. It is also the language he and Mirka use with their girls. I don't know what Djokovic's background is in relation to the English language but credit to him because he does speak it very well. I'm guessing that he learned it quite young.