A lot of things to consider. Off the top of my head:
1) did you not have a 3rd year rotation in neuro? If not, then having late exposure to neuro is fine but you will have to explain this in your personal statement & on interviews
2) IM-based letters: how many we talking? Generally, neurology requires 3 letters one of which must be from a neurologist. I've been advised to have 2 neurology letters, one of them from the neurology chair at my school and another academic neuro letter from an away rotation. My other letters are not neuro-related but strong
3) What are your thoughts on reaching out to those IM-based letter writers and telling them that you'd like their letter to discuss you being a strong applicant for neurology... also, is there a neurology chair at your school you can have a meeting with
4) on your IM rotations - did you ever have a neurology work-up (encephalopathy, stroke, etc)
5) Neuro is not competitive, but they want people who truly want to be neurologists and not using it as a backup. This ties back into the other points above. Fortunately you seemingly have an IM-based resume so it should not appear to be (at least surface level) that you are using neuro as a backup
6) Albeit not competitive, what are your scores/research/etc? Some of this can make up for a late bloom for neuro, but ultimately you will need something on your app indicating you like neurology (Student Interest group, joining AAN and attendign a virtual conference, etc)