What can do my son’s father is in prison and threatening to take my son a way when he gets out?

He is in prison. He has been in and out of prison and jail most of his life. We have a one year old son and he has never been around in his life because he will not stay clean. I won't take my son to see him unless I know he is clean. He is not on my son's birth certificate. When we broke up, he threatened my whole family and he keeps telling if I don't take my son to see him, he will take me to court and take him from me. When I was pregnant with my son, he threatened the lives of my other two kids. He is a violent person. He sleeps around and does drugs a lot. Even his new girlfriend does drugs and she was his dealer. If I don't do as he wants, he threatens me and my family. The day I got out of the hospital, he know I had to be on heart meds and he called me that night and threatened me that he was going to come out a hurt my dad and take my son. When we were to get her, he always told me if I left, he would kidnap me. I was going to move out of state but then he threatened my other two kids that my mom and dad have custody of. My mom, dad, and my nephew's live with us as well my son. He is one year old and is in good health. I take him to the doctors that he needs to go to even the doctor that is 2 hours away. He has to see the doctor because he has flop bronchiole tubes. I have never got any help from his father to raise or take care of him. The only thing he has ever gave him was a pair of shoes that are too big for him. He doesn't pay child support or help in any way. I do everything with some help from my family and I'm tired for getting threatened by him and scared that he is going to get my son taken from and need to know if he can really take him from me with his history of drug use and having felonies and being in and out of jail and prison. Is there a way I can make him stop and not worry about it anymore?