I have primary Phyiscal custody and joint legal custody. Can the father take our son out of state on his weekend?

If you have joint legal custody you share joint decision-making authority, which is different from physical custody.  Unless there is something in your agreement or court order, which restricts the father from taking the child out of town during his custodial access (physical custody) with the child, you cannot dictate what he does on his weekends unless his conduct is contrary to the child's best interest.  I strongly suggest that you consult with an experienced family lawyer in your area to discuss this matter in greater detail.   Lawyers.com offers general information and not legal advice, which is based upon the specific details of your case, which includes a review of your court order(s).  Most lawyers charge by the hour for consults, therefore, you need to discuss fees before attempting to schedule a consult to avoid any misunderstandings.  All the best,~