Med school is "difficult" because:
1) Like you said, there's tons of new information coming at you, which you're expected to incorporate into your knowledge and understanding in a relatively short amount of time. It's hard to describe. A lot of it is memorization (e.g., most of anatomy, lots of microbiology, lots of pharmacology). A lot of it is understanding concepts (e.g., physiology, pathophysiology). But there's plenty of crossover between the two. It depends on how you learn too.
2) You're competing against many others who were at the top of their class. It's not like undergrad where you could much more easily shine. Now imagine the majority are as smart as you are if not smarter.
3) For most of your clinical years, you have to read and study (as well as do everything else including have a social life and do chores like go shopping for groceries or your dirty laundry) in addition to seeing patients, rounding on patients, attending morning rounds for whatever rotation you're on, attending grand rounds, attending whatever other activities are required of you (e.g., bedside tutorials by physicians), interacting with attendings and residents and other hospital staff, often with little sleep, etc. It's not like you just go to lectures, go home, chill a bit, then have time to study (that's more true in the first half of med school). You're essentially working or apprenticing at the same time as you're studying except you're paying for it rather than getting paid for it.
4) A lot of the interactions with attendings, residents, nurses, other hospital staff and of course patients takes a certain maturity that isn't learned solely by studying books and videos. Many med students lack this maturity despite being intelligent and knowledgeable. This can reflect on you (such as in your evaluations by attendings, although evaluations can often be unfair too).
5) As for"retaining" all the information, it just comes with practice over time. You could also actively and intentionally do things like spaced repetition (e.g., Anki). But don't worry about this for now. You'll learn as you go. It might seem like you're flailing at first, but it's amazing when you start putting things together, start making correct diagnoses, etc. Don't worry, we were all there once, you'll make it too. 🙂