Can we get out of being guardian of our granddaughter

While both parents (not just one) can sign a Temporary Power of Attorney allowing someone to take the child to the doctor, register her for school, etc., just because they grant authority does not mean that someone else has to accept it.  You never needed to accept it.  Please think carefully about how you can end taking care of your granddaughter in a way which will hurt her the least.