Does asking a temporary service for reassignment equal job abandonment?

I was employed by a staffing agency who placed me at a facility. Upon entering the facility, I was faced with a high level of security which made me feel uncomfortable. I immediately left and returned to the employment agency and asked to be reassigned. They tell me they will no longer assign me claiming it as job abandonment. I don't believe this as job abandonment as I immediately reported to my supervisors.