[BHiC]

To answer your question, if someone falls, then the point is continued. If the person had missed the shot, it would have been your point, and no hindrance/let could have been called. A hindrance cannot be called for falling down or for a racquet slipping out of someone's hand. Also, if a ball falls out of your pocket, or your hat falls off, your opponent must call the let. You cannot call a let because a ball falls out of your pocket. You are correct though that it is a let the first time, but then you lose a point the second time the ball/hat falls.

This had happened to me several times, I have both fallen down, gotten back up and won the point, and also lost points because I slipped and fell.