After 20 years of possession of an affordable housing unit, would it be illegal to sell the property to a brother for a regular market rate?

I bought my affordable housing unit in 1993. After 20 years of living there, I am not longer a low income person, so I wanted to sell my property for a regular market price, following all the legal steps required by the Borough and the affordable housing authority. I sold it to my brother and he possess the title on his name with a note that says, "This property is no longer an affordable housing unit". The borough is now saying that that transaction is not legal, and because of that, they are not obligated to remove the restrictions as affordable housing unit. Are they right? Why are they saying so, if the whole transaction was made under their clear sight?