If my ex and I only had a verbal agreement, can she just decide that she won't bring my daughter over for my scheduled time when she feels like it?

My ex and I have a verbal agreement on the days we share our daughter. Nothing through the courts yet. We've had the same schedule for over a year, it's for her work schedule so I let her pick the days, an even amount of time between both parents' houses. Now she is saying if my girlfriend is at my house then my daughter can't come over. There is no reason for this except maybe jealousy. My girlfriend and my 5 year old daughter are close and my girlfriend is great with her.