I had a year in college when I had a regular 15 hr/week job while going to school full time. I had two years where I did not. I worked full time every summer. There was no relation between my grades and my job.
Don't make excuses for why your grades suffered. Take a lighter courseload, throw some classes into the summers, take an extra year to finish college, cut back on your work hours, do a postbac with all your premed science classes, whatever, but don't make excuses. Nobody abducted you in the middle of the night and forced you to be their slave. You chose the job and you chose to be a premed, and now it's your job to balance them.
In general, adcoms don't like excuse-makers. They'll give you a bonus if you're working so much because you're from an economically disadvantaged background or are a URM, but that alone won't get you in.
And in the future, do not argue that the "premeds who don't have to work have it real good." First of all, nobody gets into medical school without a large amount of work experience. Second of all, nothing gives you the right to judge people without knowing anything about them. Getting into medical school is difficult regardless of how much you work.