What are my chances of winning sole legal custody of my daughter?

I live in New York and I have been the primary caretaker of my 5-year-old daughter since birth. Her father has been in and out of prison since she was 6 months old. He was sentenced to 5 years in prison in September 2012. I can count the things he has ever provided for her on one hand. He recently wrote me a letter saying he will take me to court, because I won’t allow her to visit him. I feel as though her visiting a prison will cause a lot of emotional harm. I know he has visitation rights, but am I obligated to take her to visit? I want to seek sole custody just to secure my custodial status, and protect her. She does speak with him on the phone, and I send him pictures. Can I win custody and not be obligated to take her to a prison?