If the judge says that he cannot compel handwriting for written letters to blame another person, is this 5-6th amendment violation?

A judge does not subpoena anyone. The prosecutor and the defense may subpoena people to come to court to produce evidence or to testify. As far as letters, they are not admissible in evidence except at certain punishment hearings - such as when someone pleads guilty to the judge to allow him to decide the sentence.