How likely it is that I can have my son's father's rights terminated?

This case is being handled by Philadelphia (Pa) family court. My son’s father and I share legal custody and I have primary physical custody. He's supposed to see his father 3 weekends out of the month. His father will only take him once every 2 to 3 months!! He claims he doesn't have the money to take him or various other excuses. He does not make any effort to keep in contact with my son who is 5 years old. My son goes through a very emotional period of frustration upon return from his father, since it is never consistent. I've asked him to sign over his rights since he can't be bothered with our son, but he will not agree to that. Are my odds of winning this case in front of a judge good? Or should I not even bother going to court and continue to allow these sporadic visits. Also, I don't think this counts for anything in Philadelphia (though I'm not positive) but he hasn't paid child support in close to a year and will not show up to any contempt hearings. Shows how much he cares about our son.