What kind of motion can I file for so that my sister's boyfriend will not be allowed in our property?

Unfortunately, there is no such motion to do so. If both of you own a home together, you both have equal legal rights to the property (unless a deed or other document says otherwise), including the right to have guests on the property. Unfortunately, you have no legal right to keep him off the property if she invites him there. If she also, wants him to stay away, you both can simply consider him a trespasser and call the police to have him removed if he comes to the home. The only other way to keep him away from the property is if either of you have a protection order against him to stay away from either of you. To get such an order, he would have had to have made direct threats or physically injured one of you.