I worked in patient care for years before applying to medical school. Then I had a few odd insurance verification jobs during the summers off from college.
It all adds to some aspect of your education. The main point is that you're learning about SOMETHING having to do with the medical field, and that is building your interest in taking it further.
Don't beat yourself up about "clinical" vs "non-clinical" experience, because 99.9% of people shadowing doctors and calling it "clinical" experience are really just blowing smoke out of their ass. Watching a code being done in the ER while you disappear into a corner of the room is a LOT different from actually being there when someone actually begins to code and starting compressions, etc.
Of course, if you can shadow, don't turn it down...but do try to look at everything in your CV as something that adds to your desire to be a physician, instead of something that detracts from it.
You know you're an outstanding applicant to medical school. It's your job to SHOW that to an adcom.
I wouldn't recommend leaving your job though.