It's my weekend to have my kids but I have to work so I took them to Grandmas house can my ex legally go get them without my permission

Short answer:  Probably not.  Unless your court order says otherwise (usually by way of a "right of first refusal" clause) what you do with the kids on your time -- including who they are with and who you have watching them -- is up to you.  See the Rivero case, posted at http://willicklawgroup.com/child-custody-and-visitation/.  That said, if there is an ongoing problem, you should consult with a family law specialist.