the Davis Cup final was held soon after the Australian Open in 1984, it was probably more important to Mac to be physically ready for that. Officially he withdrew from the AO due to a wrist injury.
this excerpt says a lot about the standing of the tournament in those days(so those who just use 'quality of fields' in order to determine the 'prestige' of a tournamet are a bit misguided)
Published: November 25, 1984
John McEnroe's abrupt withdrawal from the Australian Open because of a wrist injury has touched off an intense recruiting effort by tournament officials that could cost McEnroe a $200,000 prize he thought he had already won. The officials have been trying to persuade Jimmy Connors to take McEnroe's place in the tournament, which begins tomorrow, and if Connors accepts and wins the tournament, he would displace McEnroe as the top point-getter on the 1984 Grand Prix circuit and earn the first-place bonus of $600,000.
A victory by Connors would drop McEnroe to second place and limit him to a $400,000 bonus. Because Connors had supposedly completed his Grand Prix play for the year, McEnroe had seemed assured of a first-place finish. Colin Stubbs, the tournament director, said he had established a line of contact with Connors, but admitted there was only a 20-percent chance he would be able to induce him to come to Australia. Connors's wife, Patti, is due to give birth to their second child early next month.
can you imagine the Australian Open TD today pleading with a top player today to enter the event? are we really going to pretend the '83 & '84 events had the same prestige as majors today just cause they had good fields? it took a long time for the AO to get where it was today, even in the 90s some top players skipped it(not just Agassi), it was not a given they would always be there like they are today.