I think this could definitely hurt your application, but there's not much you can do about it now and I like that answer for interviews. Have you received many interviews for the less competitive specialty? If a program is interviewing you despite the LOR then you're probably okay.
Yes the weird thing is I have around 19-20 interviews in the "less competitive specialty". Do you think that means they don't care as much as I think? Why would they waste an interview on someone they thought wasn't interested?