It's not his size. Is he smaller than Rios or Hewit? He's faster and has more stamina than players 5-6 years younger than he is. You can see this in the way he runs balls down.
It's his style of play. He doesn't junkball like Murray, he doesn't build up pressure from the baseline like Djokovic, he doesn't have Nadals assortment of insane spins, he doesn't have federer's touch or aggressive tennis (shot-making, movement and temperament).
His match against Almagro during this current Australian Open showed this. Even when Almagro was limping around in the last set, it took ages for Ferrer to take him down because his game is so defensive.
Another good match to see his weakness is Ferrer Vs. Davydenko at Doha earlier this year. He doesn't play aggressive tennis in terms of shot making and coming into the court like Davydenko does, although he is far superior to him in defense. Ferrer Vs. Nishikori at this years AO also showed that he doesn't have the ability to counter attack in the way other top 10 players do, although he did beat Kei by wearing him down and being in better shape.
He also had no net game (in comparison to other top 10 folks that is).
Mentally, he's one of the toughest competitors on tour. He just hasn't added any attacking strategies. I wonder what would happen if he had another coach that focused on this.
Disclaimer: I'm NOT a coach...
Real tennis: https://www.youtube.com/watch?v=yDqnkLJ9BtM