Yes, it's possible. Not easy, but possible. Best option is the IM program at the place where you did med school or residency. Biggest issue will be getting LORs. Talk to people you know...go from there.
The good news is that, since you want to do ID (and not something competitive) you're much more open to whatever IM program will take you.