My abusive son’s father signed over his rights and still pays child support. Can he take me back to court to try to receive his rights back?

Unfortunately, your question cannot be answered in this forum because it is not clear from the facts in your question what is meant by, "signed over his rights physical and legal."   Are you suggesting that you have sole custody? Or, are you implying that the father's residual parental rights have been terminated?  If you have a court order granting you sole custody, this type of custodial arrangement is subject to review and modification,  based on a material change in circumstances that is in the child's best interest.  Moreover, if the father is paying support, this tends to support the conclusion that you have sole custody, instead of the father's rights having been terminated, which is not subject to review.  Lawyers.com offers general information and not legal advice.  Legal advice is based upon a careful review of the facts in your case, as applied to the law in your jurisdiction.  To obtain legal advice, you must provide a complete review of your file, including all court order(s), and provide a detailed overall of facts, surrounding the father's request for modification.  It is recommended that you consult with an experienced family lawyer in your area to discuss your case in greater detail, so you can make an informed decision regarding your rights and the best interest of your child.  Best of luck~