What happens to my dad's estate if he left no will at all?

When there is no Will the assets of the deceased that are subject to probate are divided and distributed according to the state statute on descent and distribution. If the spouse is not alive the assets are divided among the children. Assets like a retirement plan typically have a designated beneficiary. Such assets pass to the designated beneficiary as directed by the deceased owner and do not pass through probate. The assets that are mortgaged should be sold and the net proceeds after paying the debts of the deceased would be divided among the children. A probate could be opened by any interested party, probably one of the children but possibly a creditor. An estate representative would be appointed to deal with the probate assets. Forcing the brother to divide the retirement plan among the other heirs is unlikely. You should speak to an attorney about the possibility and process to make the attempt.