Do we need to file the transaction of the sale with any of the state agencies if she wants to buy her sister’s out?

Who wants a quit claim deed? Without knowing more about the transaction, it is very difficult to advise you. They may be trying to avoid probate and asking you as an heir to sign a quit claim deed, but it could many other possibilities also.

If your wife's mother owned an interest in real estate at the time of her death, then the only way that interest can be transferred now, after her death, is to open a probate case. After the probate is opened a notice to creditors would be published. After any creditor claims filed with the estate are paid then the interest can be transferred to whomever. Only a court appointed representative can execute a deed to transfer an interest owned by a deceased person. If your wife's mother owned money to the state or Medicare because of support provided prior to her death then such agencies may have a lien against the ownership interest. Any transfer would have to include a payoff of that lien.