Honestly, I don't really think I did much my first semester. I didn't really know I was premed back then and was also taking a heavy load. I'm not sure med schools will care so much, as long as you find something eventually. However, have you noticed how relative 'work load' is? In middle school I used to pass up activities thinking "I don't have time for that! Look at all this homework!" But that work load was tiny in comparison to what I was doing in college or even in high school.
You need to learn to be efficient and balance your work load with other activities. You likely do have time to spare - you just have be proactive about your course work so you can attend all your ECs. Next year will be harder (with organic chemistry!), and med school will be even MORE demanding. However, we need to find time to get out and do other things. It's not healthy to sit and study all day.