If I back out of a purchase contract do I get my down payment back?

The answer to your questions are the same: look at your Agreements.  For the Earnest Money Deposit ["EMD"] as between Seller and Buyer, usually the rights and remedies are defined in the Purchase Agreement.  Custom is that there will be a term defined in the Purchase Agreement during which time the Buyer conduct due diligence and determine the suitability of the purchase; during this time, the EMD remains refundable to Buyer.  After the period, there is usually a deadline by which the EMD becomes non-refundable to Buyer except and unless in cases where the Seller breaches the Purchase Agreement.  You should have your Purchase Agreement reviewed by Counsel to determine the rights and remedies thereunder as between Buyer and Seller. As to the Escrow Holder, there is normally an separate agreement/instructions executed with the Escrow Holder defining their rights and obligations to serve as a third-party neutral and directing what is to be done with the EMD.  Most escrow holders require mutually executed instructions before disbursement of the monies.  In the absence of mutual agreement, the escrow holder holds the money for a short period of time and then will interplead (deposit) the monies into a court and take its fees prior to deposit.  The reasonableness of the fee would need to be reviewed.