Can I terminate his dad’s rights if he barely pays child support, and barely sees him?

I have 2 kids, 7 and 8 by 2 different guys. The 8 year old lives with the little ones dad who as of now has guardianship of my 8 year old. I am currently homeless, sleeping in my car and see my kids as often as I can. Over the weekend, my 8 year old went to his biological dad’s house, who is on child support. I had agreed with his wife to pick him up after 6. Well 6 came, and I called her to let her know I was 2 minutes away. She told me plans had changed as his dad was working all day and wanted to spend time with him. I refused to let my son stay there because I had plans for him. She told me I didn't take care of my son and started screaming and saying to not go to her house. I was furious, but collected. I went to her house because I wanted my son. Well, her whole family came outside including man, just standing there screaming at me to leave, and she was screaming at me because I showed up. My ex just stood there, and ultimately, I ended up leaving without my son. To not make matters worse, I left and I feel horrible and helpless as a mother, because if I called the police. I don't know if they would help me get my son since someone else has legal guardianship of him. And because I didn't want her to retaliate against my son for something I did. The thing is that my other ex has guardianship of him and he said it was OK for him to spend the night there without consulting me. Like am I invisible??! I will bring this up in court when I go for guardianship again. But I need answers for future reference.