Is it illegal to reuse an album image with my likeness superimposed on it? Ex, I want to put my picture on an old classic rock cover for humor.

You should not use other people's creative works without their permission. You could be violating their copyright. Here, you would be copying and making a derivative work of their work. Both of those actions are rights the original author has under their copyright. While there are some First Amendment rights if you want to do it for artistic purposes and not commercial purposes. However, if you intend to use it for commercial purposes, including selling it as a piece of artwork, you need to have the permission of the owner of the copyright in the original album cover.