Novak, easily. Firstly, he wasted about almost 3 years basically in the 2008-2010 period with fitness issues, he still hadn't been discovered his gluten intolerance. He broke through in 2007 at age 20 and kinda stagnated(wasted) next 3 years until he exploded finally to his full capabilities in 2011. Who knows, he might have prevented Federer winning RG in 2009, Nadal winning AO in 2009, legendary Fedal match at Wimbledon 2008? It's a "What if" scenario in tennis history...
Also, he choked/underperformed in some matches/tournaments he was expected to win. USO 2012 comes to mind(wind killed him there and he lost in a tight 5 setter in final against Murray). He has the most losses to non big 3 members in slams finals out of big 3. Federer once with Del Potro in 2009, Nadal with Wawrinka at AO 2014(because of back injury). Novak 6 times(2x Murray, 2x Wawrinka, 1x Medvedev and 1x Alcaraz).
In 2013 he allowed himself to be defeated on a hard court Grand Slam final by Nadal(USO 2013), when he's the clearly superior hard court player and he hasn't dropped 1 set against Rafa on hard court ever since. Bad form happened at a bad time. USO 2014 he lost to Nishikori, his pigeon, in a semifinal because he couldn't handle the heat well...
After winning RG in 2016 he reached a saturation point and basically temporarily retired from serious tennis for 2 years, starting from Wimbledon 2016 until Queens/Wimbledon 2018. Treated elbow which was bothering him for years(he said it started at the beginning of 2016) by just waiting miraculously to pass without surgery - Pepe Imaz, love and peace - circus, dark era. It resembles a bit Michael Jordan's temporary retirement in 1993 to "play baseball" until he finally came to senses, became hungry for success and returned in 1995. It allowed Roger and Rafa to share 3 slams each in Novak's absence from serious competition. Roger benefited especially, those 2017,2018 AO wins and even Wimbledon 2017 win might not happen if Novak wasn't temporarily "retired". Rafa would still beat him at RG 2017 and 2018 though, that doesn't change much but USO 2017 would've been tough for Rafa with Novak in there.
2020 Wimbledon cancellation. 2020 USO default(there's nothing here, he deserved it, unfortunately - just an unfortunate event...). 2022 AO absolutely ridiculous deportation and biggest diplomatic scandal in sports history and biggest scandal in tennis overall history after Monica Seles on court stabbing in 1993. He wasn't allowed to play in USO 2022 when the whole world except USA basically didn't require a vaccine against Covid, but USA did because of Biden's connections("lobbying") of pharmaceutical companies(Moderna, Pfizer) and their influence in Democratic Party(I'm not picking a political side here, just stating facts...) . He couldn't play in the US for basically 2 years and lost some weeks at #1 because of it.
So, basically he lost far more finals to inferior(non big 3 opponents) compared to Roger and Rafa,(1 against 6), he had some bad losses I think at US Open in 2012-2014 period - matches and tournaments which he should've probably won. He stagnated for 3 years in 2008-2010 because he didn't discover his gluten intolerance on time. He wasn't allowed to compete at 3 tournaments where he was the favourite to win(Wimbledon 2020 cancellation, AO 2022, USO 2022) because of Covid. He temporarily retired and messed around/wasn't serious, reached a saturation point and was injured(not so seriously) in 2016-2018, wasted another 2 years where he was still physically pretty fit(he still is at 36 actually, which is mind blowing).
There is your answer, everything explained perfectly.