FWIW, here's my perspective on dress. I did IM->anesthesiology->100% interventional pain.
I think dress should gel with what you're selling yourself as to the patient. This varies depending on specialty and setting. Dress sends a message. It tells the patient something about who you are, and what you have to offer. It can build barriers, or take them away.
For example, a plastic surgeon is frequently offering $10K cosmetic operations. What kind of dress would inspire confidence in that kind of purchase? Perfection. That's why you see these guys wearing perfectly tailored suits sometimes with French cuffs. Their offices are elaborately decorated, and even their personal office OR suites are beautiful.
An oncologist sees a very different population. What is he selling the patient? In most cases, a life not cut short. What sort of dress would give that patient confidence? Something that says "I'm a man of science. I'm brilliant, in control, and I don't miss a detail." White coats, ties, good grooming, and perhaps being adorned with a stethoscope will sell that message.
How about a doctor who works in the OR? By default it's scrubs, but that's a good thing, because crisp fresh scrubs + OR sends a message of attention to sterility, and it is the uniform patients expect. Beyond that, looking like you just stepped out of the shower helps too.
What about the ED? Filthy environment. Blood, vomit, cops around... clean scrubs are appropriate here, and maybe a coat that identifies one as an attending, being that everyone in the ED otherwise looks the same.
The ICU? Ideally a shirt and tie, because you're going to be having family meetings that need to send the message of the oncologist discussed above. But the importance of infection control and staying clean would seem to demand scrubs if you're getting physical with the patients, placing lines, etc.
And what about outpatient work in specialties that do not involve imminent life and death? Quality as opposed to quantity of life specialties I like to call them. I think the Polo and slacks is perfect. It's comfortable, sends the message that you're smart and successful, and lowers the 'me doctor' 'you patient' wall a little. My problem with scrubs in this setting is that it looks lazy, and takes away too much of your individual identity. Scrubs also set up, in my opinion, a psychological barrier that says "I'm different from you. I wear pajamas and funny little clogs to work!" Appropriate in the right setting, but not so much outside the hospital.
As for residency, it's only natural to want to wear the most comfortable clothes you can. My recommendation is to aim slightly higher than the average for your peers. After all, a senior resident in IM is doing a lot less running around the hospital than an intern, and might be able to tolerate better dress. On the other hand, he has also likely proven himself to the higher ups already, and might be able to get away with a little slack here. Bear in mind people are always going to judge you based on your appearance, and if you want that competitive fellowship slot... you need to make a great impression on everyone.