How can I change who claims my son in divorce decree?

I am in the process of doing my taxes. I am a divorced mother and when I originally split with my ex we went to a mediator and had intentions of splitting custody/visitation 50-50. Due to this, my paperwork states that we would alternate years that my son is claimed. However, since then (5 years ago), things have changed and my son remains with me most days except for every other weekend due to school and sports schedules. I feel that since I am the custodial parent and primary caretaker that I should be able to claim him each year. What should I do?