Is what the father is doing considered abandonment and can I get his father's rights be taken away?

I live in Pennsylvania with my five-year-old son, his father moved out of state to Georgia in January, moved back to PA around June/July, and moved back to Georgia soon after, where he still resides. I have legal and physical custody of our son, but his father has temporary custody every other weekend, and different holidays. Since January, my son's father has seen my son once in June (for half a day) when he came to visit other people. He has all of called here twice, and only spoke to him once for about a minute. He makes no effort to keep in touch with him in any way shape or form, and did not even tell me that he moved back to PA in June/July for a brief time, I found out from a mutual friend. The entire time he has been away from my son, he has paid his child support twice in June, and has avoided it since, while maintaining different jobs. My son has become emotionally damaged from his actions (peeing the bed, thinking everyone is going to leave him) I fear that with his father still having rights to see him, that one day he will pop back in and end up hurting my son even more. Is what he is doing considered abandonment, can I get his father's rights taken away? What can I do? Thanks you.