Can I have my children's father's rights terminated if he hasn't taken me back to court for visitation?

Terminating a parent's parental rights to a child is very difficult. It is easier if the parent agrees. If he makes no effort to see the children, why wake a sleeping dog. Plus, he is probably court ordered to pay child support which will end if you terminate his rights.