How does custody work for an unmarried couple?

Your situation sounds like a complete mess. On the unmarried issue - if you have a child and are named on the birth certificate you have to file a petition for a parenting plan and child support. If you're not on the birth certificate you will either have to file an affidavit of paternity (that puts you on the birth certificate) if it's not too late, or you have to have a DNA test or both of you have to attest to the fact that you are the biological father. The court may or may not accept the latter. None of this is a question of custody as such, it is a matter of getting a parenting plan which outlines when the child is with each parent and who has decision making authority over major decisions. As to the divorce - again custody is not the issue, a parenting plan is. Each parent will get some time with the child, same as above. How much time goes to each parent depends on a long list of criteria which the court must consider. If the mother has issues, or both of you do (see above situation) the court will appoint a guardian ad litem to make recommendations as to what kind of plan would be best for the children. I suggest you get a lawyer because your situation is quite complicated.

This is nonsensical. You can't divorce if you weren't married. You should pursue the parentage action, and stipulate to both of you being the biological parents, and go from there. Should one of you become the primary custodian that parent can proceed free (including a determination of parentage) with the County Dept.

First things first: retain a skilled family law attorney. He or she can advise and represent you, and explore this kind of question in the kind of detail that this site cannot. It is not the case that the mother generally gets 'custody.' (And it's worth keeping in mind that custody is one thing, and placement is another. It's best to keep them separate in your mind.) In a dispute about custody or placement, the court is supposed to identify the best interest of the child or children. In many counties s/he is helped by a Family Court Service, or an outside clinical social worker or psychologist. If a dispute is not amenable to mediation, the court will appoint a Guardian ad litem, a lawyer to represent the best interest of the children.