Can I deny my ex husband's visitation to our son?

My ex-husband has not seen our son in three years and has not contacted him by phone or written word in three months, he has not paid anything for our almost-four-year-old in two years. When he filed for divorce, he did not include anything about our child and I never signed anything more than the card at the post office saying I had been served. I am now (two years later) re-married and living with my new husband and our two children. My son does not remember his father and is just now learning what a father is. I believe my ex husband is planning to hitch a ride down here with his mother for our son's fourth birthday. With no legal documents in place, is it possible for me to deny his visitation? If I do, will it hurt me when we can finally afford to file for sole custody? The reason I wish to deny him visitation is a: I believe this will strongly confuse my son and b: he is not a stable man; he was removed from the marine corps for mental instability and c: he has shown no interest in our son unless a family member or new girlfriend recommended it. Can I do anything or must I allow this potential visit, and if I do deny it, will I be later punished for it if he decides to take interest and pursue custody or visitation? We live in Georgia and my ex husband lives in North Carolina.