I teach and oversee both. No contest, medicine is "harder" as the basic science curriculum is more generalized than pharmacy. There's more material and much higher time pressure than pharmacy school. I don't think the conceptual difficulty is all that much harder in medicine, but the diversity of topics as well as the other two issues combine to make medicine a fairly involved didactic. On a personal level, 20-30% of my class worked at least half-time in didactic, and 8-12% including myself worked a FT with benefits schedule as interns at our respective companies. Everyone who took summa except the valedictorian and one other guy of my class worked at least half-time (and that valedictorian was more or less unemployable as he'd was so fastidious about workflow to a fault, and the other guy had five kids and a sixth on the way during rotations).
And that doesn't compare to the two clerkship years that medicine runs. I slept at least 8 hours a day and had two days completely off without any takehome duty every week in pharmacy school and never worked past 10 hours a day at the clerkship. My medicine colleagues had something like 4-6 days off a month to take care of personal business (so it's not restful) and easily worked 10-12 hours on average per day (and had more extreme hours) with their preceptors easily more abusive and demanding than ours. I enjoyed my clerkship time as relaxing from the didactic, while medicine ramps up in that period to stress the medical student out.
In the more "humane" ACGME reforms at present, the hours are not quite as insane as when I was in training. However, there's major complaints about continuity of care issues and inculcating the attitude that medicine has standing hours like pharmacy, medicine does not and will not in practice (and the physicians that do think of in straight shifts outside primary or emergency care are being quietly eliminated for productivity reasons).
Also, in terms of major student services interventions, medicine has more problems out of its students by at least an order of magnitude than the pharmacists, and that statistic is very consistent across universities that have both. Medicine is pretty hard on weak psyches in a way pharmacy training is not.