I've signed an Employment Agreement that includes a non compete.

I am working as a Contractor on H1B at end client through a Prime vendor. he Relationship is My Emplyer -- Prime Vendor -- End Client. End Client has Right to Hire agreement with Prime Vendor and end client wants me to hire me and H1B transfer is also Done. My End Client is saying that since they dont have any interaction with my client they are free to hire me .Now my employer who doesnt have any Connection with End client is threatening me to sue me if I join the End client . As per the signed offer letter with my employer "Non-Solicitation : you agree not to provide software services to any of PRESENT EMPLOYER clients (except as directed by present Employer) directly or indirectly , while employed by Present Employer for a Period of 2 years there after." Can I join the end client , as my employer is a body shop company. If he sues me what are the consequences. Can i continue working at End Client.