First, I thought I would like patients. MS1 rolled around, I started going doing patient interviews on the wards and hated patients. Rotations rolled around, and I realized that it's just that there are certain types of patients I like, and some that I don't. I don't really like working with adults, but I love working with kids. I also really enjoyed psych, which has a LOT of patient contact. I think that every specialty is different, your interactions with patients are different, what you do on each rotation is so different that it's hard to generalize impressions from a certain instance to the entire medical field, and you don't really know what you'll enjoy and what you won't until you experience it firsthand.
Interacting with patients is also very different when you know what you're doing and you have a job to do. Honestly, I hate making small talk with people. But when I'm going to see a patient, I know what I want to ask and what I want to find out, and that plays a huge role in figuring out what's going on. You don't have to sit and chit chat, but you can't figure out what's going on until you've heard about the sx from the patient. Essentially, you can't "figure out what the problem is" without talking to the patient, so if you view it more as a means of information acquisition (which is what a patient interview is) you might find you enjoy it more, especially when you have some background and know what you're looking for and what certain answers mean.
Additionally, everyone says "oh pathology and radiology". Well, that's not necessarily true. Anesthesiologists spend the majority of their time with the patient asleep. Some surgical specialties don't require TONS of patient contact, although more so than the above mentioned. Even hospitalists generally go see their patients once per day and then spend the rest of the day following up on labs and studies. The only time you're REALLY seeing patients all day every day is in outpatient specialties such as family med, general peds, etc. Or in clinic.