Can I, as a parent, deny visitation of my child to a grandparent.

My child's grandmother is mentally ill and has outbursts in front of my toddler. She attempted suicide and even yells at my child. The grandfather is physically disabled and not able to watch my child safely unsupervised but still demands to. I want to deny visitation rights to the grandmother and only allow the grandfather to continue seeing my child during supervised visits. I'm afraid that they will sue and obtain the rights to visit without supervision.