What can I do if my husband is refusing to give our son back and we only have a verbal agreement?

I am married and was living in Ohio with my husband of 6 years. My husband and I split up and I moved to Texas with my 3 year old son and have been living here for 5 months. My husband has not seen our son in 4 months so even though we don't have custody or visitation arrangement, I didn't want him to not be able to see his child. So I gave my son to my husband for visitation for 2.5 months and I'm supposed to be able to pick him up per our verbal agreement the day after Thanksgiving. Now, he is refusing to give him back and said he wants custody and me to have visitation. We are both in new relationships and I am currently pregnant by my boyfriend here in Texas. He said I am a bad mom because I moved out of state with my son and took him away from all his family and now I'm pregnant by another man while we are still legally married. But we haven't been together for 6 months. He said because of this, he will win custody. Is this true? Can he take my child away and be granted custody? I was a stay at home mom and still am and I've never been away from my kid before. I love my kid more than anything in this world. This is killing me.