People only remember the famous GS defeats but there were plenty of times when Fed had become really frustrating to watch. I have followed Federers matches for many years. I remember the 2001 win over Sampras when it happened.
A particular period I remember Federer playing some bad tennis (not bad at all obviously, but by his lofty standards) was early in 2010 after his Australian Open win against Murray. Those tournaments before Roland Garros (Indian Wells, Miami, Rome, Estoril) really stick in my mind. The matches against Baghdatis, Gulbis, and Montanes, were a real low point for me. He seemed to completely lose that ability to be clutch and trustworthy.
So for me, I'm going to say Federer vs. Montanes at Estoril 2010 was the lowest point I can recall. Not a famous match, I realise that, but I watched probably 90% of his matches from 2004 onwards.