Real Estate. Sellers are hands off and FHA demands have cost me my rate. I wanted to back out but seller may sue me. What can I do?

The sellers lawyer stated they were going to be hands off. When the FHA made demands that were sellers responsibility they did nothing. I had to paint the exterior door trim and my lawyer, realtor and I had to work for weeks to get the city code inspector to change the house classification which isn't my responsibility as the purcahser. Then FHA made a last minute request that took us past the rate lock date. I wrote a letter stating I wanted to back out and now I get a call from my realtor stating the seller wants to sue me. What can they sue me for and what are my rights?