I also loved dermatology in med school. I would have considered it strongly if my grades were more competitive for it. There's actually quite a bit of overlap between the two fields. Check out psychodermatology. Many psychiatric conditions manifest as skin pathology, so you won't be divorced from it entirely if you do derm.
Overall, I'd do derm. I love the fact that it's visual-based, has procedures, and is easily billable.