Can my son’s father keep my son from me with me having temporary custody and is not stated in court papers?

I would need more information to fully answer this question but from what you have stated, I will give it a shot. If you have temporary custody, that should be pursuant to some sort of court decree. That temporary custody statement should spell out the custodial arrangement until a permanent arrangement can be made by the court. If you feel that your ex is violating the court order you should alert the court of this. If you know there is an arrangement temporarily in place, but just do not have a copy, you should contact the court where it was issued and get yourself a copy.