Am I allowed to take my children to California during my custody time?

I have joint legal and physical custody (50/50) that is shared with their father. Our custody agreement does not stipulate that we are not allowed to remove them from the state. We both have family in California and have both gone back and forth prior to our divorce and frequently after (I go to California with them about 1 time per month). We both currently live in Las Vegas. The only language in our decree, states that we cannot move out of state or take them out of the country without written consent from the other parent. He recently told me I am no longer allowed to take them out of state after they stayed with my sister for a few days during my custody time. I did not remove them from school as it was summer vacation and he was out of the country during this time. In the past 5 years he has never once had an issue with this. Do I need his permission going forward if our custody agreement doesn't specify so?