Can a victim refuse to testify at trial?

You've said it. Contempt.

If you were served with a subpoena, which is a court order to appear in court on a certain date and at a certain time, you are required to appear in court on the date stated on the subpoena. Failure to do so can result in the judge issuing a bench warrant or body attachment for your arrest. The police will be looking for you and when found will arrest you and eventually bring you to court to explain to the judge why you disobeyed the subpoena. The court can find you in contempt of court and sentence you to jail. If you refuse to testify, the court can also find you in contempt and sentence you to jail.

There are really two ways this comes about. First, the accuser doesn't want to testify because what seemed like a good idea (calling the cops) during an argument or to get back at someone doesn't seem like such a great idea months later when the case is set for trial. At that point, the accuser can avoid the prosecutor's subpoena. Usually, the subpoenas are served by mail (improper) and if the accuser just ignores it nothing else will happen. If the subpoena is served properly and the accuser fails to appear, the court can hold them in contempt, but CANNOT put them in jail if this is a domestic violence situation. The court can fine them and make them attend a class, but no jail. Second, the accuser can be subpoenaed to court, appear and then decide to exercise the right against self incrimination. This pops up because the accuser-witness is put in a dilemma. On the one hand, they can repeat their testimony to the cops, which may be false and if caught in the lie may subject them to perjury charges. On the other they can give new, hopefully truthful testimony, on the stand which would open them up to charges of filing a false police report earlier. Thus, if they choose not to testify and open themselves up to this, they can invoke the Fifth Amendment right against self-incrimination. In either case, it's a good idea to consult with a lawyer if you are and accuser or other witness in this situation.

There could be jail time for contempt. I have seen it happen.

That person could be put in jail for contempt of court.