Andre did complain when they played in Montreal. Federer was classy about it as usual and he didn't single Rafa out. He said that he was concerned that viewership would drop ("just serve the ball!") and he was right. Between the excessive "grunting" and time wasting from ATP and WTA players circa 2008-20016, a lot of people simply stopped watching tennis.The question is, why hasn't Fed made a big scene about this during a match? The Big 3 know they can do whatever they want, but the chair would have a easy decision to make if your options were: enforce objective rule, go against Fed.
I think this is the clear difference between Fed and Rafa. Rafa will do anything to win. Fed will fume silently, and try to shut him up by beating him. If the roles were reversed Rafa would complain in the first set every time they met.