My father has been deceased for 17 years. I received a check from an insurance company but, it was made out to the estate of my deceased father.

You can only cash it by opening an estate.  If the check is for less than this will cost, you can let the funds go to unclaimed property with the state comptroller and in four to six years (it varies by state), show that you (and sny siblings) are the heir(s).