Would this be considered a threat?

Yes I would say it's a threat if he is not family he can not lay a hand on you. Even if he was family he still can not threaten you. You have rights also. Turn him into the cops. Call Victims Of Family violence and get in a report so If something Does happen to you This person will be Questioned. Maybe the reason your mom never done anything is because she is also afraid of him.