If he still has rights in the future, can he just try and come get him whenever he wants?

My son is going to be 4 in March. His father has been out of his life for the majority of his life. When he was around he did not have a relationship with him. He has had no contact with him for at least 2 years now and refuses to give up his rights. He doesn’t want to pay child support or medical and said that he isn’t comfortable meeting his son or being around him, but still refuses to sign over his rights.