Can I send my neighbor a letter stating the retaining wall between us is 5" on my side and only 3 1/2 on his. He believes the whole wall is his.

Yes, but the letter will not move the wall.  You can sue to do it.  Check your survey from when you bought the house and see what your surveyor said.

You'd better- or eventually he will own the property under the doctrine of adverse possession. In New York, you can legally acquire your neighbor's property if you enclose or improve the disputed property and your action is open, notorious, hostile, and under a claim of legal right, for 15 years or more.  However, as you state your neighbor is "highly volatile", the letter should come from a lawyer- not you. The letter should include a copy of the deed, survey and property description as proof of the property lines, and if no recent survey is available, you may need to order a survey of your property.