What can I legally do if ex won’t allow vacation with son as punishment for his bad grades?

I'm the primary parent of my 15 and 17 year old sons and their dad has them every other weekend. They both have passports. We've traveled overseas several times the past couple of years. My 15 year old and I have been planning for months to go to Asia at the end of the year. My ex initially agreed and now says no due to our son's bad report card. He has a whole summer of punishment planned plus is dictating our vacation at years end. The more I try to reason with my ex, the more spiteful he becomes. We haven't bought the plane tickets yet but I want to buy soon while prices are cheap. Our child custody agreement has no stipulations on vacation and our travels would not interfere with his visitation. Can he stop us from leaving the country?