I made a contract on a house which was to be null and void should I not get approved for a mortgage. Got denied & buyer wants my deposit.

I found a house and when I went to make an offer, there were already 2 offers so I offered $5000 over asking price. I was pre qualified for a conventional mortgage, however, when my offer was accepted and I put my official application in, it turned out my DTI ratio was too high and I was declined. I told my realtor what happened and that I could not get the house. He pressured me into applying for an FHA loan which I never intended on doing but I did, and again, got preapproved. When my realtor let the seller know the situation, they said they would only accept if I would agree to pay any and all repairs that the FHA inspector found. I did not agree to this, so I terminated the contract which my realtor said I had the right to do because of the section that said if you apply for [convenional] financing and get denied within 40 days of this contract, it is null and void and the deposit ($1500) will be returned to the buyer. The seller now wants to sue me for the deposit. What can I do?