Is there any way I can withdraw my support for sponsoring my husband and preventing him from renewing his permanent visa?

It will expire at the end of the year? I am currently undergoing a contested divorce and we had a child together. It was not in the best interest for my daughter to live with him as he was never there when the baby was born or even care for my daughter such as providing basic needs (e.g. changing the baby diaper, etc.) I don't quite recall all the forms I filled out to get him to US, but I have not gone to the interview to support him to get a green card yet. At this point, my priority is to have the full custody of the child and limit the number of days he can visit, if having any visit days at all. I spoke to my attorney who received a letter from my husband's attorney to have a standard plan of seeing the child with very minimal child support. Please let me know if you can help me with your own opinion and experience on how best to handle this situation. Should I also hire an immigrant attorney as well?