Murray was no slouch. He's made 11 Slam finals and beat Djokovic in 2 of them. He would be a worthy top 3/4 player in any era.
It's true that 34-year-old Fed wasn't exactly the best top rival, but he was still pretty damn good for his age, probably the greatest 34-year-old of all time. He was still ahead of the field and continued beating Djokovic fairly often.
There were still some dangerous top 5-10 players out there; Wawrinka, Nishikori, Tsonga, Berdych, Cilic were pretty solid competition. Overall I'd rate Djokovic's 2014-15 field and Federer's 2005-07 field roughly similar. Neither were peak Big 4-era strength, but history has certainly seen weaker eras.