Is there such a thing as a miss trial when it comes to a custody/divorce trial?

For there to be a mistrial, some improper matter must have been brought before the trier of fact that was too serious for the trier of fact to dislodge from their mind.  In family law, the judge is the trier of fact so I can't see how that could happen. Were you thinking of something different?