Can he deny visits because he doesn't want the specified monitor to do it?

My ex-husband and I went to court last year to modify custody because I moved. The court order states that he has full physical custody and that I have supervised visits on June 15 and 16, and July 21 and 22 with my brother. Then it says both parties are to work together to expand time share. I, not realizing exactly what it said have been continuing supervised visits with my brother. I have had about 8 visits and talked on the phone with my child 5 or 6 times a week. About a month ago, my ability to get in touch with my daughter has halted with no explanation. I tried to set up to visits with excuses given to me as to why he can't drop my child off. I have continued to call daily and have texted him twice to have her call. After 18 days of no response, he finally said that I will not be getting any phone time with my daughter saying that I did something at the last visit that caused me to lose that privilege but still hasn't told me what that was. My brother group texted both of us saying that he hasn't shared any information to either parent as to the visits. Now, my ex wants to change the supervised visits to the court supervised visitation office and has put my visit for next week on hold until he figures out if that's legal. With the order that we have presently to which both parties agreed to, (not ruled by judge) do I still have supervised visits when only 2 months were specified? Can he withhold phone contact? I am really stressed over this and just want to contact my daughter. Thanks for any advice.