I always think people get this wrong.
Golf is an enormous sport. A very popular sport. Nearly anything can be done "for sport", engaged in through serious competitions. Bass fishing is a sport, competitive eating is a sport. Poker can be a sport. Bowling is a sport.
I think what people want to say is that golfers are not "real" or "elite" athletes. I'm not agreeing or disagreeing with this, but it would make much more sense. "Track and field" in the Olympics is (or at least was) officially called "Athletics" and that makes sense since the skills on display in track and field match what most of us think of when we think of elite athletes - freakish speed, strength and/or endurance. Further, those traits are absolutely essential to success. Sure, there are buff golfers who played other sports and are indeed good athletes, but, it is not absolutely necessary in golf to be a great athlete to be successful.
But, many of us generally associate "sports" with "athletes" instead of more accurately "sports" with "sportsmen" (which would include athletes and non athletes depending on the sport).