depends on the elective - for something like rad, anesthesia, and path i doubt it would matter too much. but for any advanced medicine, pediatric, surgical, or subspecialty elective the problem i see is that you won't have the knowledge foundation necessary to even keep up during rounds.
for example, you decide to do a cardiology elective before doing internal medicine. well, at my school during IM we cover ACS management, chest pain workup, EKG basics, hypertension management, etc. These are basic things to a cardiologist and your elective will be spent catching up with what you should know from general medicine instead of focusing on the higher level issues that a subspecialist focuses their time and energy on. i think the reason the traditional med school curiculum includes these core rotations is because they are essential to anyone practicing medicine, and essential to the education of a doctor.
from a residency issue i can't see how it would matter, other than the risk of poor performance for the reasons stated above.