Federer reached 5 French Open finals and only lost 4 of them because he was playing against the clay GOAT. It seems incredibly obvious that no one deserved to win a French Open (who hadn't already won one) more than Roger.
It even feels very fair in terms of the Rafa-Roger dynamic. Rafa owns Roger on clay, but Rafa's style of play is so physically demanding that you expect him to have some issues from time to time. Rafa wins almost every French Open, which is quite just. However, Roger just kept showing up, doing great, and giving himself the opportunity by making the finals, until one year Rafa didn't bring it (for whatever reason) and got himself knocked out. A well-earned grand slam, I'd say.
Think about it this way: in any other era, Roger making the finals of Roland Garros 5 times would probably have netted him at least two French Open titles.