How can I get out of a real estate contract as a seller?

I was threatened with a potential law suit for a breach of contract. The home was inherited from my dad who died in March 2021. My husband wanted to find out if we could add an addition on to the home because we wanted to move there. I called the city in April. The building department returned my call. The man said that the property was zoned industrial and that we could not add on in any way. Told me other reasons why I shouldn't live there too. Since we could not add on I decided to sell the house. I got an offer and signed a contract. Went down to the city to get the inspection and asked out of curiosity if I could add on to the house. The lady told me yes. It is zoned M2 Non-conforming. I asked are you sure? She told me to talk to the building director to make sure they were not giving me false information again. So, I called my realtor and backed out of the contract. He was ok with it, but the buyer and other realtor may sue.