One of the things I've observed about physicians is that, relative to the other professionals in the hospital, they tend to have more work to do relative to the time they have to do it. Physician salaries, especially in a primary care setting, eat up a huge chunk of the payroll and hospitals/practices make back the money by scheduling physicians as many patients as they can conceivably handle. On the other hand most non-physician hospital personel, like most office workers everywhere, do a very good job but eat up a significant portion of the day chatting and screwing around on the internet. They're not useless or lazy, they are in fact working more often than not, but they have a certain amount of slack built into their schedules so that they only work every minute of the day on rare, 'perfect storm' type days. On the other hand most physicians are scheduled so that they work every minute of the day on an ideal day, and even the most inevitable deviations from the plan push them into the realm of unpaid overtime.
Now as always there are exceptions to every rule: I have seen ER nurses that never stop moving and I have seen docs that spend huge chunks of their days screwing around. However, in general, I wouldn't take offense when a physician is all business while everyone else is saying good morning. The doc is just doing what he needs to do to leave on time.