Can I lose custody of my 2 kids for dating a man who was charged with 2nd degree assault?

I have been divorced for almost 2 years now. It was an ugly process through another state. I was able to get our decree adopted. He has remarried and moved closer to us. I found a man who I as well as my children adore. Could their father take me back to court and get custody of the kids? They are 9 and 6. The main problem here is that I was the one he assaulted. There is much more behind the story. He is great with the kids. They have nothing negative to say about him. I feel he is a positive influence on them. He is working towards his career path and working 2 jobs to be able to do so. He plays games with them and treats me with respect in front of them (and not). He helps me around the house, we both help with their homework, chores, manners and all. My ex has made my life hell and it hasn’t stopped since the divorce. Should I be worried that he can take me to court and get full custody? Right now, he has every other weekend with them.