As far as I can tell, the answer to that is no as long as you have a legitimate reason. Today i moved my Northwestern interview because it was during finals week, and they didn't have a problem with it.
Hmm, what about schools that have rolling admissions? I asked this when rescheduling with Northwestern (which apparently does admit on a rolling basis) and was told that yes, it's "a little better" to interview earlier. I wasn't sure what to make of that, but I had to reschedule anyway so..