Is it possible for my father to write a document giving my mother custody of me and both of them sign it and notarize it without court?

It would be best for your father or mother to hire an attorney to prepare the necessary documents. No, the way you worded your question will not work. They have to sign an agreement and then the judge has to sign an order based upon the agreement, most likely.

In order to answer your question, you would need to advise whether your parents were ever married to each other, and if so, whether they are still married, whether there is a divorce pending or if they are already divorced; if they were never married to each other, whether there is either a paternity judgment or other court order determining custody of you; if they are divorced, whether there is a divorce judgment determining custody of you; and whether there is any legal action currently between them.