can a firm force you to sign over your share of the sale of your home to pay legal fee if not they will file papers in court they no longer represent

Its hard to answer based on the information you provided.  If the firm represented you and they are holdilng onto money you are entitled to, they might try to resolve the matter by making you let them keep the money to satisfy the fee.  The firm can't "force" you to agree to that unless they have filed some kind of collection action and gotten a judgement against you.