What can I do if I already filed an I-30 for my sister and niece but my niece wants to start studying here in the US?

If your niece files for a student visa, it will not affect your I-130 for her. However, the existence of a filed I-130 is likely to result in denial of your niece's F-1 visa application. Every applicant for a non-immigrant visa must prove, to satisfaction of the U.S. consul, that she has no immigrant intent, which means that she does not intend to take residence in the U.S. and intends to return to her homeland after her stay in the U.S. If the applicant cannot prove this, her visa must be denied. Proving absence of immigrant intent is seldom easy; in your niece's case, it is exceedingly difficult because the I-130 petition you filed shows conclusively that your niece does, in fact, have the intent to take residence in the U.S. So, unless your niece can show to the U.S. consul that she has some circumstances that would compel her to return to her country after the course of study in the U.S. and wait at home for her immigrant visa, filing for F-1 is going to be a waste of time and money.

Look for other ways to bring her here, such as an F-1 student visa.