Can I get sole custody while the father is in jail?

As of right now I have physical custody and joint legal. I would like to have sole custody of my children. They haven’t seen their father in a year and I was just informed that he is in jail. He is going to be serving at least 3 years in prison. He has 5 children 2 live with him 1 was taken away by the state and then my two. His wife and he keep trying to get me to take my children for a visit while he is still in a county jail. But I won’t take them I do not think jail is a place to take kids. And since they stopped seeing their father they have made complete turnaround in school and at home. Their father has made a number of complaints against me to CPS and they never open a case due to there is no abuse or neglect. My children are very well cared for and very happy kids.