Like yourself, I've wondered about this exact same issue.
The problem, I think, comes when you get into smaller sized towns. If you're practicing in my town (Columbia, MO) which has a population that's lower than 100,000, you'll be practicing in a health care community which has the resources to support you.
For instance, 80-90% of oncology is outpatient, now. This is good, but those 10-20% who are inpatient need someone to take care of them. I mean, if you have 30 of your patients in the hospital, and are still trying to carry on with your outpatient practice, you need some back up. You'd also want someone who specializes in Critical Care Medicine. You don't want to have to manage a Neutropenic Patient who's in the Unit, while managing other outpatients and all your inpatients.
That said, it makes sense then to practice in a community where you can have a group of oncologists to back you up. Someone takes the inpatients for a month, and you have a critical care doc to man the unit. You're not going to find that in towns much smaller than 30,000 people.
And no one can do it all--all the time. It's rough. So, oncology is a practice which is geared for bigger cities, but by no means do you have to practice in a place which has greater than 100,000 people. It's all about what you want your practice to be and how you want to taylor it.