Can my daughter's father bring up a non violent criminal past of my boyfriend to take her from me?

A judge must determine the best interest of the child when awarding custody to one parent or the other.  The Pennsylvania custody law lists 16 factors that must be considered when the judge makes that determination.  One of them is the history of drug or alcohol abuse of a party or a member of a party's household.  If your boyfriend's criminal past involves drug or alcohol abuse, it will be considered.  The 16th factor listed in the statute is "any other relevant factor", which means that anything else may be considered by the judge.  Also, if your or your child's father or anyone who lives in either of your households has bee convicted of a serious crime that is listed in the statute, including driving under the influence of alcohol or controlled substance or driving after imbibing alcohol or controlled substances, a judge must consider the person's conduct and determine that the person does not pose a threat of harm to the child before making any custody order.  In my experience, if your boyfriend was convicted of shoplifting five years ago, a judge would not find that to be of much importance in custody.  However, if he was convicted of manufacture or distribution of drugs 6 months ago, this would weight more heavily in the judge's decision.