I think it depends on the school and the type of work you did. For example, I used to do behavioral therapy with kids with Autism, and a couple programs thought that was really interesting even though it had nothing to do with vet med. Did it help me get in? Ehh probably not a significant factor. Some schools like to see diversity in what you have done and might ask how your past experiences have prepared you for a career in veterinary medicine.
As mentioned by SnowyRox, Colorado has a section where you indicate how many hours per week you worked, how many units you took, and you GPA by semester/quarter. That tells them what your workload was like, how it may have affected your GPA, and how you are able to manage your time. Schools like Colorado understand that someone who has to work full time during college may not have as good as a GPA as someone who didn't.
Overall, I don't think it matters that much, but there's a section on the VMCAS for other work experiences other than vet/animal. I think you should put it down, but it's probably not going to have a whole lot of influence in the decision process.