Can landlord withhold security deposit for early termination of lease?

My wife and I sought to break our lease a month early in order to move into a more affordable apartment. We spoke with our landlords about moving out early (just on the phone, there was no written correspondence to prove this) and they seemed amicable and quickly found some new tenants. We were never given a copy of the lease we signed, and when we asked about our security deposit, they replied that they would not be returning it, saying that that lease agreement states that if the tenant move out early, they would withhold the whole amount of the security deposit. We are leaving the apartment in excellent condition and the Landlords will not lose any amount of money in terms of rent, so my question is, do my wife and I have reason to claim for our security deposit back, or because of the clause in the lease agreement do we not have any claim?