Variety of reasons:
1) You do two "intern years," with the first one (medicine) usually being much more inpatient-heavy than the medicine categoricals
2) Often much higher patient acuity and constant volume. Constant ED consults, constant stroke alerts, etc. No one can wait. Our consult service sees probably 25-30 new patients and 30-40 follow-ups a day.
3) Very high patient volumes on primary service, more in line with surgical services. Neurology services typically don't have a "cap" like medicine services, so the stroke service may have 30 patients, half of which are ICU-level, with 2-3 residents.
4) Call is often brutal (and often a 24-28 hour shift). My personal record as a resident was seeing 34 patients in a 24-hour period, which is far from my program's record.
5) You have specialty knowledge about a field no one else in the hospital knows literally anything about, and so will constantly get called either about meaningless things, or questions to which no resident could possibly know the answer. Unfortunately, neurology knowledge bases are typically very limited even among other physicians, let alone nurses/APPs, so you get called ALL the time with questions.
6) This is institution-dependent, but many non-neurologists consider neurology to be a "subspecialty" of medicine and that neurology residents = medicine residents. This means you can't reject a patient for medical complexity (something any other service in the hospital can do), generally can't get much help from medicine consultants, and have more disposition fights than any other service. That patient in DKA with a sodium of 120 who looks septic, but also had a 10-second seizure? Guess who's going to get asked to admit the patient? This sounds like a minor issue, but can be very frustrating.
7) Also institutional-dependent, and becoming less of a thing these days, but due to all the reasons above, duty hour violations used to be a big issue at many places.