Can the mother of my daughter only allows me to see my daughter but not with my wife and other kids? Is this even legal?

I have a temporary order. My 12 years old daughter refuses to see me according to her mother. This has been going on now for 6 weeks. I have asked her mother if I can speak to her and the response was she doesn't want to talk to me. I am married with other kids and her mother tells me that my kids are a bad influence to my daughter. No clue what those as she refuses to tell me. Instead, she wants a sit down talk which was willing to but we want a mediator present to facilitate the talk and she refuses. The mom is willing to let me see my daughter on my scheduled weekends but only me and not my wife and other kids. She said she consulted an attorney and told her that this is the right thing to do.