Was my boss in the wrong?

Yesterday my boss abruptky and out of the blue , told me my maxi dresses, that I have been wearing to work for the past 10 weeks mind you with no problem, offended him and distracted him and had probably offended and distracted customers. He would need me to dress more conservatively, He did this while we were alone and he was staring at my chest. I said nothing except ok. As I said I have worn these for the past 8 weeks, our office is very casual most wear shorts and jeans, and my 350 lb co-worker wears her " beach dresses" with no problem from them. I began to wear the dresses because my office stays 8-10 degrees hotter than other rooms in the building because they will not fix the air. Also it bothers me because I was made to feel like I had offeneded someone when I work around people who hurl racial , socio-economic, and slurs about anyone who doesn't believe in their political views. I am bothered by what happened and had to ask were they in the right?