You are right to be concerned. Their Agreement should be limited to inventions related to their business (check, as the language may already be in place). If this company is in the same line of business as your athletic company, you have a conflict and need to choose one business going forward.
Otherwise, they should be able to accept that IP related to your athletic company will not be assigned over as a result of your employment. You don't need specialized legal language to achieve this goal, but it should be made plain and clear as a written modification (again, if the language is not already in place) in the agreement before you sign it.
Now is the time to avoid a foreseeable problem.
Good luck,
Todd
Answered on Jan 13th, 2014 at 2:03 PM