may i stop brother from entering our home to visit dad?

My father can no longer make decisioons, I have been POA 5yrs now & caring for my dad 24/7 alone, my brother upsets my dad & me when he visits, hes done some awful things to us, as POA, can i insist that my brother see my dad anywhere but our house as my brother feels he can see my dad 24/7 at our home and doesn't show up when you say ok but only wants to show up when you say no, my brothers medical license was revoked, hes faking a 9/11 survivorship to get ins., hes been fraudently writing my dad off taxes, gave my sisterinlaw hpv & had kids with other woman, stold money from late mother etc etc, he upsets my dad & me, as POA can i keep him away from my dad or atleasr away from our home and can i tell him not to see my dad when my dad is sick ot not up to it, w/o legal coonsequences that hes not seeing my dad when he wants to ?