Four years is a good AVERAGE to receive your BS/BA in. However, that implies that some people do it in three years and others do it in five. not everyone does it in four, that is just the average. as for switching universities...the only problem you are going to run into there is explaining why you switched in your interviews and the process of trying to make sure all the transcripts from each school are sent in on time when you're applying is very annoying.
I had taken summer courses at a community college and my last semester i had to take an addition 3 hours at a different college that had became partners with the community college i had attended in previous years. needless to say, when i sent in my transcripts those two college's had the grades on one transcript and AADSAS didn't realize this for 3 weeks until i called them and asked why my application had not been processed. I was very angry at them. it posponed my application for a total of a month and a half.
i'm sure you'll be fine though.