During the entire stretch of Federer's approximately 8-year prime, he won an average of six matches (out of a possible seven) a year at the AO, collecting four titles along the way. During those same years, when Djokovic played the AO, he won an average of three matches a year, collecting but one title. They split their two matches, but there were only two because only one player showed up late in tournaments, and it wasn't Djokovic.
If there were failures to come through in that era, they weren't Federer's. He owned that era, which included plenty of fields where Djokovic wasn't up to the challenge.
Now is that fair to a young Djokovic? I'd say irrelevant, but it's Djokovic fanbois who want to bring it up, so whatever.
Both guys have had their primes, and faced each other in fields when the other guy wasn't in his. Federer won during his prime, and fought his way deep into draws when he wasn't in it any more. Djokovic won during his prime too, but came up like a mug most of the rest of the time.
Were they consistent rivals in that era? Well, they were .500. But I'd say it was more like the rivalry between the hammer and the nail.