My daughter and her 4 month old son live with my wife and I. The father and my daughter are not married. Can I keep the father of the baby from.....

You are entitled to keep whomever you want out of your house, but the real question is how you are going to enforce that.  If your daughter lets him in when you are not there, then you will have to address that with her, it's not really a legal issue.  If the baby's father feels that he is being denied access to see his child, he could file a petition for custody or visitation with the court.  Because he does have parental rights in this regard, it would be best to work out some arrangement with him that you are comfortable with, such as supervised visitation.  This answer is given in accordance with the laws of Virginia and may not be applicable in any other state.  It should not be construed as legal advice, as that would require a more thorough analysis of all of the facts involved in a specific case.