How can I take away his right without getting my kids involved?

My ex-husband has only seen my kids once for maybe 45 minutes in the past going on 7 years. He has no contact with my kids at all or me. My phone number has been the same for 7 years. I've remarried. I've been with my husband for 6 years. My kids know my husband as dad. They don't know they’re biological as dad at all. They know him as uncle. They came home from my ex-in laws calling him that he's behind in support a lot. I've asked him to sign over his rights. He says no but still has no contact with me or my kids. They are to 7 and 8.