Can I refuse visitation if the father was in a serious skydiving accident and is unable to care for the children himself?

You can't just refuse. The question is if he is able to interact with the children, even if someone else is doing the physical caretaking, then the court would likely order it, if you withhold and he files a motion for contempt. The way you need to think about this is the children's father was hurt. Seeing him, if he can interact with them, will ease their burden (as well as being his right). If on the other hand, he is on so much pain killer that he can't interact at all, then a reasonable solution is to have only brief visits (when he can) and resume normal visitation when he is able.