What should I do if I am accused of harassing people and being a threat to others but I have never done anything?

I believe that a former employer has gone to the police, stating that I harass people and am somehow a threat to others around me. During my time at this employer, I wasn't written up once throughout the whole nine years I was there. Now whenever I am in a work assignment, I will hear people being told not to harass me, and that if I harass anyone, the police should be contacted. I am not doing anything to warrant this attention. If ever I sit next to a female, she will always be asked quietly if I was harassing her. The police or a former employer must be giving every work assignment I have a very negative commentary of me. It seems that with this negative commentary every work assignment wants to rid of me even though I am not doing anything to warrant this negative attention. This is limiting my ability to provide financially for me and my family. What should I do in this situation? It is going on right now at the temporary assignment I am currently working at.