Can my mortgage lender use funds from my old escrow account, without my permission, to pay down the balance of my old mortgage upon refinancing?

My old mortgage was with Wells Fargo. I refinanced that mortgage again with Wells Fargo under the VA IRRRL program. After closing, Wells Fargo sent me a check for the balance in my old mortgage escrow account. The amount of the check seemed too small so I called Wells Fargo. They said that they erred when computing the old mortgage balance so, upon, closing, they used funds from my old escrow account to cover the difference when paying off my old mortgage. They say they did not need my permission to do this. It seems to me that Wells Fargo took money out of my pocket without my permission. After all, I didn't refinance only part of a mortgage. It was my impression that my escrow was a trust account to hold my money to only pay for property tax and home owners insurance. Wells Fargo is refusing to do anything to straighten this out and return my money. Do I have any legal recourse? We are only talking about $500 and change. Please advise. Thank you.