You are a little unclear as to where you are in the process, but if, as it appears, you have already appeared at a contested hearing of some sort, then no, generally you cannot file a peremptory challenge, except "for cause," and the facts you set out in your question would not qualify.
For what it is worth, consider the possibility that you may have mis-perceived the message -- no matter WHAT your situation is, it is perfectly possible that the judge in question HAS seen worse. Unfortunately, we see the most horrific cases of abuse, even murder; it is possible that the judge was simply trying to commiserate. But either way, you should probably focus on the merits of the issues that are pending.
Answered on Aug 01st, 2014 at 1:30 PM