Is it illegal to keep my child from her father without his name on the birth certificate?

I broke up to the father of my child a few months ago. I didn't say he couldn't see her, I just said it had to be supervised. He won't file anything through court and he's admitted to me he's been using pills for months at beginning of break. He's telling me I'm breaking the law by not letting him take her because I don't have hard evidence to prove he is taking drugs. He also has no job, lives with his girlfriend my child has never met, no car nothing. I honestly don't feel like my child is safe with him. He also disappeared for 3 months and then popped back up demanding to pick her up on a school night. Am I doing anything illegal by doing this? I haven't filed child support because I work from Monday to Friday before court opens until right after it closes.