If I quit my employment with express temp agency can I work for an employee who wants to hire me?

What does your contract with the temp agency provide?  What does your prospective employer's contract with the temp agency provide?  it is likely that one of the two, if not both, prohibits what you want to do, or requires you and/or your new employer to pay a finder's fee to the temp agency if it hires you directly.  If there is no such contractual restriction, and if you won't be using any trade secrets of the temp agency in your new employment, I don't see any legal restriction on it.