Harassment by building handyman

I live in a 3-story apartment building. The responsibility for putting out the garbage is the building handyman. When the garbage was not put out at the curb for two weeks, I called my landlord and told him. He sent the handyman to take care of it. Last Wed. night when I came home from work the handyman stopped me and said "why are you getting me in trouble?" He made several untrue accusations about me calling the handyman in the next building, which I had never done. Anyway, the handyman in my building then said "your husband is dead and you're all alone now, right?" I felt threatened by this remark and called my landlord to let him know. He agreed that those remarks were totally uncalled for and said he would give the handyman one more warning (without mentioning my name) and then he was gone. I am afraid the handyman is going to know who complained about him and will retaliate. Do I have a case of illegal harassment?