How do I get our earnest money back on a property we have not even been yet financed for?

We have been going through the buying process at a new build site with the builders lender. We have picked out the items to be put into the townhome. There has not yet been a final construction meeting, and their financing for us has not yet come back approved. We did give 3500.00 earnest money, and were told we would probably not get it back when we told him we have to back out. He said the lumber has been purchased for the building, even though two units are still unsold. We didn't have a realtor. Our mistake. Also, I asked him yesterday if he had a form we could sign stating that we were officially backing out, he said he did not.