How am I protected as an unwed mother when my 4 year olds father threatens to take my son because I'm choosing to move out of state?

I hsve 3 boys who all live with me. My 4 year olds father is threatening to take my son because of my choice to relocate been to Dallas. He states that I have 2 already I don't need the third with me. The boys are VERY close. We do lots of family things together from movie nights to game night.. I invest alot of time in and with my boys. They're all honor roll students even though the 4yr old is in head start he's excelling well. My concern is, how I can proceed with this move without the father trying to say I'm kidnapping his son? There are no court orders in place, no child support as he.. Kind of takes care of things... Actually he pays 200 to the daycare only.. Which is the part time amount because he refuses to pay more than that. It's hard getting clothes and shoes from him so I just buy them myself. But what limits if any do I have? My 4yr olds father is his grandmother (his father's mom) gets him for weekends. I've offered to continue the weekend visits by meeting him but that