It depends. You say the receiver was moving in closer to the service line at the time. Is it possible that the receiver just coudn't get out of the way in time (rather than hitting it deliberately knowing it was out)? Or that it was not a reflexive sticking-out of the racket on the part of the receiver?
Basically, if I thought it might have been involuntary on the part of the receiver, I would claim the point. That's the price to pay for moving in on the serve. But if it is clear that it was voluntary and that he was just clearing the ball then I would not claim the point. Even though the rules allow you to, I think it would set an unpleasant tone for the rest of the match.
The funny thing is most people I play will not try to touch the ball, voluntary or otherwise. They know they lose the point if it touches them so they try to avoid it. (unless there is obvious circumstances that require them to catch it, like a hole in the fence with a cliff behind it and man-eating tigers, etc....)
The rare person I find who cant help themselves always seems to be the person who sets the unpleasant tone for the match.
Most matches that I play in though, even if just for "fun", consist of people who care about winning by the rules. We watched tennis on TV as kids and played high school tennis and making up some game outside of that is not really "fun".
However if I find myself in another crowd (permanant court time leagues, mixers, etc...), I usually dont say anything because they tend to play by their own rules anyway.
As far as being a better person for not calling it, Im sorry any reasoning you come up with can be countered. So you were hitting the ball out?? So what??
If you were going to hit it out and they hit it in mid-air and THEIR ball went out, you wouldnt give them the point. Most people would just say "whew!!", they saved your point.
It's unfortuanate for them and good for you, but that's tennis. Get over it, just move on....