It is all about overhead. You can have a "small practice" if you plan properly. This means:
* Having one employee that does everything for you (except billing).
* Seeing only 5-6 patients in the morning and 5-6 in the afternoon.
* Having a small office with only two exam rooms, with a really, really low rent.
Internal Medicine might be more amenable to this type of practice than family medicine. As an general internist, you could advertise yourself as a "Chronic Disease Specialist" and deal primarily with diabetic patients and those with chronic heart and lung diseases. Then you advertise "by appointment only" and in so doing, you basically keep out the "riff raff" (acute conditions, like colds and runny noses and other boring junk).
So it absolutely can be done. A lot of docs say it can't, but it still can. What is difficult, however, is to have a BIG solo practice with tons of exam rooms and 10 employees. With reimbursements so low as they are, you'd have a tough time keeping your lights on.
When you are in medical school, a lot of docs will try to tell you that it is impossible. And then they will offer you a spot in their group with a starting salary of $120. Basically... they are just trying to bring you in under their roof so they can work you like a dog and make mad cash off of you. Don't fall for that. As your own boss, you can do that well AND set your hours, vacation time, etc.
I hope this encourages you.