Is my non-compete with my ex-employer actually enforceable if Im not doing anything wrong?

I work in insurance- I worked for a company for a year before my office was sold and the person who took over at that time then made me sign a non-compete. Very broad stating I couldn't work in insurance in almost any capacity within the same county for 1 year from leaving. So now, 3 years later I have decided to leave, and have an offer to work down the street. I did not take any customer lists and do not plan to solicit any of my old cstmrs; but, it's a small town and is likely ex customers may find me in which case, to be very careful I will have someone else in the new office assist them in the sales process or tell them to wait. So my ?is, if I am not actually soliciting anyone or doing anything to hurt my ex emplyr other than just actually being at the new office, is it enforceable and does he actually have a case just because I am there? Also, my potential new emplyr did call my ex boss to ask permission to hire me (small town) and he said yes but did tell him I had a non cmpte.