Some of the posters address the main issue about tennis: those athletic kids that play american football and soccer don't want to play it. This is obviously for a few reasons:
1) It looks both easy and boring when they see other beginners playing (and frankly, the same with pros on TV to an untrained eye).
2) ANYONE can do well at the sport, no matter the level of athleticism
3) Beginners barely work up a sweat at all
4) most beginners can't keep a long rally going, so much time is spent picking up balls
5) There is no physical contact involved at the beginner level
6) The scoring is complicated and counter-intuitive
7) Most importantly, it's just really difficult when you start!
Tennis and various other racquet sports are unlike most other ball sports (or any sport for that matter) in that while many other sports revolve around catching, throwing, or kicking--all of which are naturally learned movements, and are therefore simple to perform the basic object of the sport (ex. american football, some of baseball, basketball, soccer, rugby, water polo, volleyball, the list goes on)--hitting sports like all of tennis, racquetball, squash, and badminton, and some of baseball and cricket, require a learned and refined skill set for the most part. Since both objects, the ball and the hitting device must be moving, the timing, hand-eye coordination, and balance all make the sport very difficult to progress to high level in. With the non-hitting sports, there is always a direct contact with the ball, making the basic motions of the respective sports easier to do relatively well in. In short, with any given person, it's a lot easier to have fun AND work up a sweat in the non-hitting sports, like american football and soccer, than in the hitting sports, like tennis and squash.
All that being said, the reason that tennis looks so easy is essentially because it's so hard at the beginner level. Beginners end up barely moving their arms or their feet, and the ball moves very slow, since it's so incredibly difficult to find the balance between hitting hard and hitting with control. The combination of using your arms, hands, legs, and feet all at the same time makes it ridiculously hard to enjoy yourself AND get a good workout.
I suppose the solution to all these unfortunate facts would be if there were a lot more high level tennis players out there, so that when "laypersons" see the sports, they think, "wow, that looks really hard but also really fun." The problem with that solution is that when those "laypersons" try to play, they think, "I'm light years behind those players I saw; it'll take forever to get that good," so then they just give up altogether.
Alas, it seems the beautiful sport of tennis is trapped in a vicious dilemma of popularity.