One thing people miss when judging has-been pro athletes is that these people are on the very top of what they do. It's usually a very specific field, in the grand scheme obsolete, so they have engaged people drooling all over them and the indifferent majority indifferent, but at what they do, they're very much the best of the best. They were working on it since they were little buggers, and they know that no matter what they do, no matter how much money or, hell, even intelligence they possess, they'll never ever reach this level of accomplishment again in another field.
The road to mediocrity, or even sub-mediocrity of the post-retirement life of a one time top athlete is a scary proposition for many of them, because nothing satisfies as being the best. It's not even ego. It's the dopamine kick they can't stimulate through walking the dog among millionaire mansions.