Is there a good and effective way to ask for a retainer that implies suspicion on their practice as one of the reasons we are firing them?

I hired a law firm to handle a loan modification. They referred us to a per diem paralegal who I come to find is involved with another law firm as stated on her Linkdn account and after finding a plethora of complaints against that particular law firm with regard to loan mod scams. I even found a summons officially filed in NYS as they were being sued for fraud and misrepresentation by 11 individuals on one summons. We met with her twice, no papers were filed to our loan servicer, nor any financials submitted to her for submission as our financials were not in good order at the time. She spoke in circles each time we met. No answers. Told us to stop paying our mortgage and to not speak to the bank. A flake all around. Should have walked away, yes. But we thought she was just the paralegal we would meet with initially and we never got to meet the attorney we had been there to see. I paid 1500.00 towards a 2500 retainer. I want at least 1000.00 returned as she will likely bill be for the two appointments we had. Is there a good and effective way to ask for a retainer that implies suspicion on their practice as one of the reasons we are firing them? It can't possibly be as simple as asking for my money back.