Someone married me for their papers, now I want a divorce. Will I have to file or will the divorce be automatic when he leaves?

A divorce is never automatic.  It's very important that you file and have him served before he leaves as service is mandatory and the divorce or annulment cannot go forward without service. If he would agree that he married you for papers, you may have grounds for annulment which would void the marriage as if it never took place.