How can they label this house a drug house and use this as a legal premise to harass anyone who comes here?

I have been in and out of trouble my whole life. Over the last 4 or 5 years, I have stopped living like an idiot and started that process of growing up. I have successfully discharged my parole. I have paid off fines. I have gained full time employment and held that job continuously for those last 4 years. I’m 44 years old, and have been an addict since I was 13 years old. I mind my own business and want to be left alone. The local police, some I’ve known for years some who don’t know me at all, have taken it upon themselves to continuously drive slowly by our house, taking note of any faces they see and or cars parked in front. They will then wait for anyone they recognize to leave, and pull them over stating that their probable cause to stop them is because they were seen leaving a known drug house. While conducting their searches, they badger and interrogate my friends asking why are they there, telling them things like stay away from me and I’m bad news or they know what I was doing. These things are not true, and I feel that what they are doing is affecting my reputation as a functional law abiding citizen. I feel that I am being slandered and I know what they are doing is not right.