Whether Phelps juiced hardly matters. If he juiced, you know everyone else in the pool juiced. PEDs or not, dominating every aspect of a sport so thoroughly is unbelievable. Would be another matter if Phelps spent years suing and shaming and threatening people. Doesn't look like he's done that.
If you play by the rules some might consider you a *. If that's a problem then, by all means, give away as many points as necessary for their approval