If my mother application for tourist visa was always denied because she overstayed before, can I sponsor her for the green card to be a U.S resident?

Generally, a U.S. citizen age 21 or older may sponsor a parent to become a Permanent Resident through "consular processing" in the Immediate Relative visa category, and this is true even if the parent may have previously overstayed a visa. There are exceptions to this general rule, however, such as in instances where a parent had overstayed a visa for more than 180 days (and become subject to a 3-year bar to re-entering). It would be wise for you and your mother to consult with an immigration attorney who, after learning all of the relevant information, could advise about immigration eligibilities, options and strategies and who then could offer legal representation in the often complex application process.