Can my ex legally prevent my boyfriend from spending time with my kids? (NYS)

My boyfriend comes over and watches my kids on occasion when I have to work, or run an errand in the evening. They are very comfortable with him, enjoy spending time with him, and frequently ask for him to come over more often. He has been a part of their lives for almost two years, and a part of mine for longer. My ex and I have joint custody, but they spend a majority of their time with me. My divorce was very high conflict and my ex absolutely hates my boyfriend. He found out my daughter was sick and stayed home from school, and my boyfriend came over to watch her since he had the day off and I did not. My ex freaked out and is saying he can prevent me from leaving my kids with my boyfriend, and can dictate who they have contact with.