My school does a mix of in-house and NBME. I generally prefer NBME because I study better/more efficiently from textbooks and board materials, and it's nice to get an idea of where you stand score-wise compared to most med students so you know where your weak spots are and where to focus come dedicated time. Generally, the questions are better-written but there have been some crap ones. But there are def issues with NBME exams as well.
- You can still get random minutiae as others have mentioned which were not necessarily emphasized in lecture or review materials, whereas in my experience the minutiae from in-house exams were at least discussed to a reasonable extent
- Some of the NBME questions are based on out-of-date guidelines (looking at you, OBGYN)
- You do not have the chance to review the test or protest questions after an NBME exam, so you have no idea what you got wrong and you can't really learn from the exam
However, for a few courses we had really well-done in-house exams. Questions were super well written and fairly similar to NBME questions, the subjects tested were extremely fair and representative of what was emphasized in lecture, and we had the chance to review the exam afterward. And to be fair, my school does a really solid job of focusing lectures on topics that are relevant to patient care and/or the boards, and avoiding the esoteric "let's talk about the nitty gritty details of my research" stuff. If we could have had all of our exams be in-house at that level of quality, I think I would have preferred it to NBME.