Can my grandson be taken away from me if there is no parenting plan in order?

My daughter and her ex boyfriend do not have a parenting plan. The father lives in WA and we live in Kansas. If I bring my grandson to WA to visit family, can the father take my grandson? Other details that might be helpful. My grandson was born in California, my husband is in the Army. That is how we ended up in Kansas. My daughter is not financially stable to live on her own so she is with us at this time. A year ago when we all visited family in WA, my daughter brought my grandson to his father's home for a visit. The father's sister took my grandson into the house and the father closed the door on my daughter. My grandson was only 5 months old and was still breastfeeding. The plan was that my daughter was staying there during the visit. After knocking on the door numerous times, she called the police. The police came and she was told that because there is not a parenting plan in effect, the father has physical custody of the baby right now and does not have to return him. Definite panic set in. Fortunately, the father's mother came home and gave my grandson back to my daughter. We have not been in WA since that time. My daughter might be going for the holidays but will not be meeting the father unless they have a parenting plan in effect. I would like to go sometime after the new year. With my daughter being in school and working evenings, I was planning on taking my grandson with me. He will be 21 months at that time. I have been told that if the father finds out I am in WA with my grandson/his son, that he can call the police and have my grandson taken from me. I do not want to put myself nor my grandson in a compromising situation. Thank you.