Can I terminate the father's parental rights and how would I go about doing that?

I live in PA my daughter is 3 yrs. old I have attempted to let her father be a part of her life on 3 different occasions but he continued to fail. He has never paid for support and I have never filed because I was advised that if I let him be then maybe he would let me be. 1 year ago he filed for visitation it was ordered that he could have visitation as mother agreed. In that time he had not once asked me to see her. In over 6 months I have not heard from him. I want to terminate his rights because he tends to come and go when it works for him and she doesn't need that confusion. She has an entire family who loves her. I am a single mom, I work a full time demanding job and my time not at work is spent with her dating hasn't been my priority but now I'm being told because I'm single I can't terminate his rights. My parents are like her second set of parents why can't they adopt even just one of them? Single parents can adopt why can't I protect my child as a single parent? Courts don't want to make children part orphan but she already is. My daughter doesn't know who he is at all. What can I do? I know that this won't be easy but it's not fair to her to have a man who can come and go as he pleases what will that teach her, I read the research on children. With inconsistent father's it's just as if not more concerning than that of a child without a father. I'm not someone who is just trying to get rid of my ex I wish everyday he loved his daughter but he has never even sent her a birthday card or asked how she is doing. Please help!!!