Name matters but it's not like it's stanford or berkeley.
@EverStriving did point out that UCSB would probably have better research opportunities but if you are determined to do well, you can keep a high GPA at Cal Poly SLO, form great relationships with your profs and learn the material very well. Then with the proper study regiment you can score well on the MCAT. Other than making sure you have the appropriate ECs (volunteering, shadowing, etc), the most important factors for med school admissions are your GPA and MCAT. The GPA is not equal across all schools and disciplines, but a high one does indicate a level of scholastic aptitude and work ethic. The MCAT gives a view of how well you're prepared to jump through the necessary hoops - while "bigger" schools do tend to have applicants with higher scores, it's ultimately up to the individual to put in the work to get the higher score.
So overall I would say go to the place that would allow you to be the most successful student. If you believe that UCSB will provide you with that then by all means attend and make sure you are able to keep a competitive GPA. If Cal Poly seems to fit the bill better - with proper preparation, it won't stop you from getting into a solid med school. Though that med school may not be in CA due to the insane applicant pool lol.