Should I be worried about the biological father getting custody of my child from me because of my record?

Once I get enough money I plan to have a DNA test done on my 2 year old cause it was a 2 time fling and I know who the real father is. His wife was going to buy an over the counter DNA test and supposedly now they have talked to a lawyer who advised them just to have me take them to court for DNA and child support. They have also threatened that once he is proven to be my baby's biological father, they are going to try and get custody from me because neither have been arrested and have their own home and stuff. They are trying to say that their evidence against me and my husband is that I had been arrested 2 times in 2008 for drugs (which I successfully completed a drug therapy program and no other trouble since) and my husband's arrest record, and that I don't have my oldest three children (which I'm currently in the process of writing a letter to the judge to petition the courts to regain custody from my mother). I have also had a couple people try to involve department of children and families a few times and they have came out and I did what they asked and closed out the accusation. I’m also involved with the Healthy Families Program as well. Also my husband and I are in a rent-to-own home of our own as well. Should I be worried? My healthy family’s worker said not to be especially besides the above things, they have never even met my child and no court would take him away from me cause of the trauma that would do to him.