Can my ex keep my son away from my boyfriend who has a record?

I have a boyfriend who committed a misdemeanor of domestic violence against his girlfriend's child. In the police report it states that he hit the child with excessive force when he was disciplining her for writing on the wall. After talking to him about what happened I believe what he said. He said that he was the father figure in the child's life. He also said that he did not mean to hit the child hard and that was his first time ever hitting her in the three years that he was with the girlfriend. He says it was a mistake he regrets everyday of his life and one that he will not make again. My question is can my ex keep my son from being around him or take me to court over this issue? What should I do?