My husband pays child support. Ex wife wants to sit down and write a custody agreement and get it notarized. We live in pa. Will it hold up in court?

yes and no.  it's certainly evidence of what type of custody arrangement the mom and dad want.  but it is NOT an Order of Court, so the court  often won't actually enforce it.  and certainly the police are even less likely to enforce it.  to be safe, your husband needs a custody order that is signed by a judge and filed with the court located in the county where the child primarily resides.