I think this could definitely hurt your application, but there's not much you can do about it now and I like that answer for interviews. Have you received many interviews for the less competitive specialty? If a program is interviewing you despite the LOR then you're probably okay.