My uncle has POA of my grandmother. He wants me to give back my house keys to her house and start making appointments to see her. Is that illegal?

I use to be her caregiver and have a full time job. After 8 years my grandmother kicks me out of the house cause my Uncle who is her POA told her to. Now he wants me to turn over my house key so ONLY him and my aunt have access to the house. I need to start making appointments to see my grandmother. He informed my grandmother to tell me that he doesn’t work around my schedule. I need to work around his schedule. Yet this is not about him this is for my grandmother. He leaves her alone for hours to go drinking and gamble. He makes her cry. Do I have rights to see my grandmother without his permission?