can the state come after me for what happens at their dads house?

Me and my two children live in Franklin Ky. I recieved a call today from state social workers concerning my son. My son is six years old and just went to his dads for the weekend. When he went to school he told his teacher that he saw his dad and his dad's girl friend naked and humping he asked them to stop but they didn't. The school and state workers are expressing concerns over this of which I am to. I don't know what goes on over there at their dad's house other than what the children tell me. In the past the children have mentioned this situation before of which I have approached their dad about acouple of times in the past. Each time he has denied it saying the kids are making up stories. I expressed to him that it is not approiate behaviour and to please stop. Its been at least a year since the children have mentioned this senior to me. Can the state of KY come after me for something I have no control over?