Can my son's dad get custody if I already have sole custody and he hasn't been in my son's life?

I have full sole custody of my son. I got custody not long after his father and I separated. His dad moved away to Florida. He has only seen my son a handful of times. My son knows who he is but doesn't connect with him. He doesn't know anything about my son. My son has medical conditions which needs to be attended daily. I own my own home and I have a full time job. I take my son to every doctor's appointment. His dad doesn't have his own home. He lives with his family and he works from 7 pm to 7 am. He has told me before that he doesn't think my son needs to be on his medications. My son is 6 years old and doesn't want to talk to his dad. On the rare occasions that his dad has called to talk to him, my son would lay the phone down and continue with what he is doing or he would hang up on his dad. He doesn't know what to talk to him about. My son acts like he's bored with the same questions that his dad always asks and when my son answers him, it's like an automatic response. When my ex filed for custody recently, he didn't even have all the correct information. He put a phone number on the paperwork that I haven't had in 4 1/2 years. The sad part is that he has my correct phone number. He didn't even file in the correct district. He filed in a district that I haven't lived in for over a year now.