Can my ex get custody of our child because my boyfriend is a felon?

My ex and I never married. Paternity was established. The baby is 16 months old. My boyfriend has a B felony for domestic battery against his last girlfriend. That was in 2010. He has hit me once and that was last week. My ex boyfriend has proof of all of this because I left a voice mail when my boyfriend hit me. My boyfriend does yell a lot. But not at my son. He leaves lots of voice mails on my ex's phone about how I can't afford milk or diapers for my baby. But we only do that to get him to pay child support. My ex said he was having a hard time paying it all the time. But I think he makes it up. So my boyfriend tells him we can't get milk or diapers. But unfortunately there are voice mails of this. And we also only have a one bedroom house, so my son shares a room with us. And my boyfriend's grandchildren are over almost every day. My ex says our son shouldn't be around a felon. He also says that he thinks my boyfriend will eventually hurt our son. Now he says he's filing for custody. Does he really have a chance? It isn't like our son's the one being hit or yelled at.