Sunscreen! Personal mystery solved, as I've had this issue as well on some of my whites. Also couldn't figure out why some of my white wristbands were getting so stained over time.
If anyone knows of a sunscreen that stains less than others, please share. Guessing they're about all the same, though.
"Ever tried. Ever failed. No matter. Try Again. Fail again. Fail better." - Stan's left arm