Does a Serbian college student, currently attending classes in the USA, need to return to Serbia in order to renew his visa?

Your visa (the page in your passport with your photograph on it) can expire without the need for you to renew it if you plan to stay in the US as a student.  The I-94 card, which for students generally is notated with "D/S" meaning "Duration of Status" allows you to stay in the US indefinitely as long as you continue your studies and have a current I-20 from your school proving you are enrolled.  Your visa does not control how long you can stay in the US, that's what the I-94 card is for.  You do not need to renew your visa unless you want to leave the US for personal reasons and then re-enter to continue your studies.   The visa just get's you into the US.  Once you are here, the I-94 card is what allows you to stay.