Who is responsible for my penalty?

My tax accountant filed my returns late and told me that his submitted excuses would be accepted by the IRS.[Death in family, hospital visit and more.] I paid my taxes and have now received a penalty. I think that he is responsible for my penalty. We entered into an agreement back in February. I have not contacted him about this matter yet.