Can I still apply for H1B or green card within the USA if my home country cancels my passport?

Generally, all aliens in the U.S. *must* have a valid passport at all times: it is one of the conditions of every immigration status. You also must have a valid passport and attach a copy of it to every application you might file with USCIS or the Immigration Court. If USCIS becomes aware of the fact that your home country cancelled your passport, it will want to know the reason. You might be wanted back home to submit to a criminal investigation, or to answer summons to a court, or to surrender yourself to penal authorities to serve a sentence, to pay child support or a judgment imposed by the court, etc. If so, the U.S. government will likely revoke your status and ask you to leave. However, if you are in trouble with your own government because of your race, ethnicity, religion, political opinion, or belonging to a particular social group, you can ask for an asylum in the U.S. or for withholding of removal.