The money thing is always going to fluctuate, so I wouldn't let that influence you a whole lot.
The "jack of all trades and master of none" is a legitimate drawback to the field. You will have a more superficial knowledge of each body system than a corresponding specialist in the field. A gastroenterologist will know more about the GI tract than a primary care doc. A cardiologist will know more about the heart...etc...
To me, there is no shame in this. As a family doc, your input is just as important and at times you will have valid reasons to override the treatment recommendations of a specialist. Some examples:
Ophthalmologists generally want all of their patients on high doses of vitamins A, C, E, Zn, Copper to prevent macular degeneration. Its up to the family doctor to know when that's appropriate, because for many patients it isn't.
Nephrologists seem to approach hypertension as though the body is a life support system for a pair of kidneys. As primary care, you have to recognize which patients need to be on beta blockers (as opposed to the all time favorite of nephrologists: the calcium channel blocker) or otherwise to improve all patient outcomes, not just reduction in proteinuria.
Actually, there are far too many examples too list where your knowledge in primary care will override the recommendations of a specialist. You just can't be a sheep. The bottom line: there is nothing inferior about your knowledge base in primary care if you do your job right. While your knowledge is not as deep, the broad scope of practice often puts you in a better situation to help your patients make the right decisions. If you work hard and back up your decisions with good medical evidence, then your specialist peers will be forced to respect you. So, to me, the broad scope and lack of depth in some areas, it is not a big deal.
The biggest drawback I see in family medicine is that you often don't have the reward of seeing your patients get measurably better. They come to the office, you give them prescriptions, and then they leave. If they had an acute problem, and they got measurably better, you'll never know it. You'll have your chronic patients, and you'll get them through crisis to crisis, but the chronic illnesses will remain. Most patients don't get measurably better. This is a stark contrast to a surgeon or a hospitalist whose patients, generally, get markedly better as a result of their interventions.
I did IM for 18 months before switching to FM. I do kind of miss the times when I had a patients acutely dyspneic from pulmonary edema, I gave them lasix, and their respiratory statuts markedly improved in front of my eyes as their foley bag filled with clear urine. You rarely see these kind of results from anything you do in Family Medicine, or primary care in general, outside of the hospital. To me, that's the biggest drawback in primary care.