Will I get denied for green card renewal if I've invited my mother to the USA and she stayed here with expired visa?

Your inviting your mother to the US and her staying with an expired visa has no relevance to your ability to renew your green card. Green card renewal is more concerned with whether you have committed crimes, but the main purpose for which U.S.C.I.S. requests green card renewals is to maintain the security of the cards. Over time, counterfeiters have been able to duplicate many of the features of the green cards, and so the agency has placed time limits on cards that it now issues and puts in a number of new security features periodically.Due to the limitations of the Lawyers.com Forums, Alan Lee, Esq.'s (the "Firm") participation in responding to questions posted herein does not constitute legal advice, nor legal representation of the person or entity posting a question. No Attorney/Client relationship is or shall be construed to be created hereby. The information provided herein by the Firm is general, and requires that the poster obtain specific legal advice from an attorney. The poster shall not rely upon the information provided herein as legal advice nor as the basis for making any decisions of legal consequence.  

No, your mother didn't have to have an invitation from you to visit.  You are not responsible for her decision to overstay.  Moreover, they won't connect your mother's record as a non-immigrant with your record as a permanent resident.  They only perform criminal background checks when re-issuing cards.  You'll be fine.