What can I do if i see my son experiencing disability harassment in our townhouse community, from the adults?

My son is 12 with numerous neurological issues, including processing issues, Tourette's disorder, and Autism spectrum. Everyone knows him and knows of his disabilities. However, some of the adults think it's ok to yell at him--for riding his skateboard too close to their house, for cursing (yes, he has that kind of tourette's), and basically, any time there is trouble in the neighborhood, someone points a finger at my son. I'm tired of it. I have spoken to these particular parents, I have even had to file a complaint when one of the fathers told my son "I'm going to get you, kid". I think this one is the one spreading venom about my son to others. We have dealt with kid bullies in the neighborhood in the past-I had to threaten a parent with harassment charges before it stopped...but what do i do about the adults? There is a condo association, do I send a letter to them? To the offenders individually? It's two families that I know of that are doing this.