Just tell him you're not taking no ****. Don't fight him, just tell him straight that you're not happy with him, and sort everything out. If not, you should go with everyone else. A few punches directly in the face, a couple of uppercuts, and a roundhouse up his ass should solve this problem...