Straight up his point. The ball bounced on your side of the net and then hit the ground again before you touched it. Remember it doesn't matter what it hits once it hits in bounds on your side - the point is his when it hits the netpost, the side fence, your tennis bag, or his side of the court.
If you play by the rules some might consider you a *. If that's a problem then, by all means, give away as many points as necessary for their approval