Of course it matters, blue books have no residency programs so you're not even able to make yourself known to those hospitals for the purposes of the match during your rotations. It doesn't mean you won't match but it is worrying that if all of them are like that than your school won't be able to help you secure any decent elective rotations either. During COVID, a decent chunk of programs are no longer taking IMGs for elective rotations so the connections your school has becomes even more important.
You haven't named your school but it isn't a big 3 school given the fact that you don't have any teaching hospital rotations. If you haven't started this program yet, I'd think long and hard if it is something you really want to commit to.