Can my ex stop me from going on a weekend trip with our child?

My ex-husband and I have joint legal custody and I am the physical custodial parent. The only thing the parenting plan states is that visitation as both parties reasonably agree. I want to take my son out of state from Friday to Sunday to for a much needed getaway. He has our child the 3 days prior as he is off those 3 days and works Friday - Sunday. However has to in his 2nd day of work demand be stop by to give our son a hug and kiss. I will be going with the new person in my life as well and he has a girlfriend. However he has been very controlling and even trying to dictate who I can and can’t see and is now saying I can’t go if we all stay in the same hotel room. There is nothing of this nature in the divorce decree nor does it say I can’t travel with out his person. It just states visitation as we reasonably agree and his demands are unreasonable and if I say no he becomes very enraged.