What would happen if we moved out of state without telling her because it is highly unlikely that she would call bother to call, or ask to see him?

My husband and I would like to move out of state. We have physical custody of his son whom we had since he was eight months old until now he is eight years old. His bio mother (They were never married) lives one city block away from us and has only visited him about six times since. She has not seen him at all in a year and the visit before then was four years ago. Our children go to the same school as her others and we see her every morning. He does not know who she is. She does not bother to say hi, visit or anything. The last time that we tried to get full custody and do a step parent adoption she had not seen him in five years. She got a lawyer and fought us saying she changed and wanted to do everything she could for him, despite what the court appointed lawyer for his son said about her. Our lawyer said, the judge denied us, saying that she did the best thing a parent could for a child by staying away while she was getting into trouble "jail, drugs, children and youth" Since the ruling two years ago, she seen him only once, and has not asked called or even stopped us at the school. If we move to another state and she calls, or Facebook (her choice of contact), months to years from now, what could happen? It is not going to change or disturb any visitation because there is none. There is just random visits every so many years and since she does not own a car we are always the ones to take him, (children and youth says we must be there or a responsible adult, chaperoned since she is involved in yet another case). If we bring him to these seldom and random visits would there or could there be trouble for us? The move would be from PA to NC.