What do I do about a non custodial visitation without a court order?

Would it be advisable to let my son's non custodial father to pick him up and keep him over a weekend if he's never even met the child, who is 8 months old, and for someone who has threatened to keep the child at one point in time. Can I be "punished" by the court for not allowing him to take him? I don't want it to look like I'm denying him visitation, because I have offered for him to come to my home to visit him, just not take him anywhere. I feel like that is the best for my son, at least until there is a court order for visitation. I fear that he may try to keep my son and I wouldn't be able to get him back until a court date because he's the type that would do something like that.