Can my husband adopt my child without bringing biological father to court?

My child is 4 years old and the biological father has never paid anything for child support. He has only seen the child one time for 5 minutes when she was 2/3 months old and never heard from him again. She was the product of him forcing himself onto me and I do have an apology note that is very indirect but of him apologizing to me for doing so. My husband has raised my child since she was a year old and after another year, I had moved and changed my number because after two years of the biological father showing no interest, no contact, nothing, we decided to raise our child on our own and blocked him from social media. We have seen him in public and he acts like he doesn't even know us. My husband wants to adopt her but we do not want to get the biological father involved because we feel that if he gets brought into it, he will attempt to alter the story like he has to other people many times and between him forcing himself onto me. He has been in rehab recently, and his family is very aggressive causing me to go into hiding during my pregnancy. She turned 4 a few months ago and my husband is ready to adopt as soon as possible. Biological father is not on the birth certificate and paternity has never been established nor requested. He constantly is telling people very extreme lies about me, has come to my work harassing me and making threats to me, and asks other people for pictures of her but has never once made an effort directly to me to be in her life. Are his rights automatically terminated because of the two years of him knowing my phone number, contact information, and address and never maintaining a relationship with her?