I was dismissed after freshman year because of academic problems, but I was able to successfully appeal the dismissal and get reinstated. However, several semesters of academic probation and the academic dismissal are on my permanent record and I mentioned it on my AMCAS and on secondaries.
I have had multiple interviews, including at some top schools, and have an acceptance so far. MD-only, btw.
Yes, the academic dismissal will be a negative. The more important thing is how you recover from it afterwards. I maintained a 4.0 with 20+ credit hours/semester for nearly 3 years straight after my dismissal to fight back from around a 1.5 GPA to nearly a 3.6. Interviewers were very impressed with this and I think my story of hitting rock bottom and bouncing back into competitive range intrigued adcoms and got me interviews/acceptances.
It'll be hard, no doubt. But not impossible. Good luck!