My ex husband found out that I had my parents babysit our son while I went a date one time.

The short answer to the question you asked is "no."  You don't provide some significant details (like the actual time share for the child between you and your ex, or how often is "once in a while"), but (1) you are entitled to live your life and associate/date with whoever you please, presuming no danger is being created for the child; and (2) it is generally considered a good thing for grandparents and other relatives to have some opportunity to spend time with and develop a close association with a child (again, presuming there is nothing about those relatives that is a danger to the child).  If you are not gone so much that an objective observer would have concerns of abandonment of the child, it shoiuld not be a concern.  That said, if your decree includes a "right of first refusal" (you do not say so), then your ex should be given the opportunity to take the child if you are gone for whatever the triggering amount of time might be (typically 4 hours to one day); much depends on the terms of the current court order.  But your ex does not get to control your social life.