I guess they must have no memory. My kid is starting out and he barely runs any faster and he's the fastest kid in his class and he can't even run 3 miles. I don't know how fast everyone else was when they started, but I know my first mile wasn't all that fast either. And you will probably improve sooner.
Unfortunately, there just aren't a lot of people over age 40 taking up jogging. I think it's highly commendable, but it's going against the trend (which is mostly down and out).
The prime age for runners is mid 30's. Meaning that across the board you will find more solid runners at that age than any other. Distance running is a very mental sport which a lot of people don't realize.