When can I deny vistiation to the non-custodial parent

I am the sole legal/physical custodial parent of my children. Non-custodial parent has been in/out of jail for the past 5 yrs for drugs. Currently, court order states visitation is to be determined between both of us. There is no specific schedule. He was having frequent visits with them every Saturday from about 11-7:30pm. This is following his release from a prison stay that lasted 7 months. He was released in May and got picked up again in November (this year), for possession of a controlled substance, violation, and driving on a suspended license. He was released and the case was dismissed. My question is, based on our current order, can I not agree to allowing him to see the children? I have since discovered that he has been driving them, which we agreed he wouldn't due to his license being suspended, and he is telling my 6 and 5 year old to lie to me about it and I am worried about their well-being. I don't trust that he is clean and my kids are struggling with his inconsistency.