Being a physics student, I think there's a reasonable chance you'll like o-chem, honestly. Once you get through learning the naming algorithms, it's a lot more based on pattern recognition and internalizing relationships-- you can build the whole subject up from a few basic concepts. As a chem major who was pretty frustrated with what seemed like the arbitrariness of a lot of gen chem, it was a good turning point for my interest in the subject.
Biochem is a bit more disorganized and memorization-intensive, but there's also a chance you could pick it up from MCAT study materials (e.g. Kaplan + AK Lectures) if you are generally a self-driven learner and are really opposed to taking the class. That said, it's a significant portion of two sections of the exam these days, so more extensive practice working and gaining familiarity in the problem space really can't hurt.
Good luck in your journey!