Context, context, context. His comment was specifically in relation to his H2H matches with Federer, not his overall record, and the H2H scorelines were indeed very competitive i.e. "not far off".
Still, how the big points are played is a large part of what separates a champion from a pretender. Milos is still in the pretender category as far as challenging for a major now.