How effective are DMCA take down notices in terms of preventing piracy on identified infringements?

Dear Questioner, Interesting question. You state you've discovered an unauthorized distribution of your copyright (registered?) work but rather than seek to shut it down, you may permissively allow it to continue. Pre-supposing that you could shut it down if you wanted to, making the conscious decision to allow the torrent could potentially have a negative impact on your lawsuit depending on how the issue was framed it is an argument you can control and need to explore further with a lawyer. This isn't "unclean hands" that is a legal doctrine your question is using inappropriately. It would be better to contact the bittorrent operator and offer a license or some other arrangement. DMCA notices will, generally, get a bittorrent taken down, unless the website in question is one that flagrantly disobeys copyright regulations (e.g. PirateBay), or, perhaps, is operated in a jurisdiction far outside the sphere of law of the USA and Europe. Regardless of whether you permissively leave up the torrent, you can still sue for the unauthorized distribution. Your situation sounds complicated. If you're going to sue, you'll need to see an attorney anyway. I recommend you pose this question to your attorney, along with all the other pertinent facts. This message is just general information. For a good, solid legal opinion, you need a full consult where the facts of your situation will be thoroughly explored.

It sounds like you are trying to be a copyright troll. If you play such shenanigans, yes, any court case is likely to fail based on acquiescence, estoppel, laches, or unclean hands. Don't be a creep. If you want your material taken down, contact the website and insist on it. If you want it to stay up, then don't plan to sue. It sounds as if you need to be advised by a knowledgeable, honest lawyer. Don't get involved in predatory behaviors, such as copyright trolling, because karma will come back to bite you sooner or later.

You are obligated to protect your copyright and that includes not making it available to be infringed.

The take down rules do not affect your right to sue someone who infringes your copyright. Sending take down notices is an effective way to stop the widespread piracy that the Internet makes possible because doing so is quick and easy while suing individual infringers is so expensive and time-consuming. Failing to send a take down notice to the owner of the site where the torrent file is located may be poor judgment but it does not subject you to the defense of unclean hands from third-parties infringes.