This has happened to me before. Sometimes, as you did, I kind of forget to watch the line closely. Or the word "Out!" gets stuck in my throat. By the time I get my act together, it feels too late to call the ball out, so we play on.
I don't feel sorry for your opponents at all. I mean, it's not like you whistled the return past them for a winner because they believed the serve was out. They had a sitter.
I probably wouldn't have agreed to a let unless the serve was way, way out, like a foot.
-- Random Error Generator, Version 4.0
-- Master Moonballer