Can a judge deny unsupervised visitation due to who the father lives with

my ex-boyfriend lives with his mother, who constantly is drinking and has lots of people over and it tends to get out of hand. I don't want my daughter over there without me. I allow her father to come over and see her but while he visits he sits on his phone and hardly pays attention to her. I want to know what a judge might do about this. The grandmother demands that I let my daughter come to their house but without me. She constantly tells others that I do not allow her son to see his daughter and that my daughter is not allowed to see her other side of the family. I have on multiple occasions invited them to my house to see her but they always come up with an excuse of why they can't come. So I have not allowed the father to come over as often because it is to stressful to deal with it all and then I am not the best mom that I can be when I have to worry about all that. I am not sure what I should do or what a court would do about it.