I have full time custody of my daughter through a verbal agreement, can I move with her to another county?

I live with my 15 year old daughter. Since she was 5 years old I have had full time custody, through a verbal agreement with her father. For the last 6 years I have been struggling to afford a home for us, and her father only pays a small amount of child support. This month my daughter and I will be facing possible homelessness for the second time in less than one year. Because of health conditions and a lack of sufficient finances, I want to move us to a city where we could have an easier time surviving. I can't allow my daughter to live with her father because he uses dangerous substances and is rarely home. But he refuses to let us move away from him. Even though our custody agreement is not ordered by a judge, do I have the right to move away, since I have cared for her full time for 10 years?