What can I legally do if my ex wants longer custody time of our children than stipulated in our divorce agreement?

I was divorced a few years ago, and am now remarried and living in a different state. My ex and I have joint legal custody of our young daughter, but the agreement stipulates that I have physical custody, and the ex can take her for the summers. Now he says he wants to have our daughter for both the summer and one school semester annually. He says that he will take me to court to force this to happen, and that he will win because the daughter should live with her real dad more than 1 month each summer. But she is in a stable home now, has lots of friends, and is attending a good school. Taking her away for more than the summer would be very disruptive. Do I have anything to worry about from his threat to take me to court? Thank you.