Empathy and competence are far from mutually exclusive. If anything, the finest surgeons I've worked with are also some of the most empathetic, especially those who are truly world class and have patients flying in from all over. Here's a key point: brilliance and competence are the gifts that allow someone the luxury of time for empathy and rapport. I've definitely noticed a trend where the less empathetic docs are generally less competent as well, much like the truly gifted superstar med students also tend to be the kindest most well rounded because they have that luxury.
What OP is describing is a very real aspect of clinical growth - the need to balance rapport and empathy with efficiency. The answer is not to be a douche and blaze through your clinics or else you will quickly see your referrals dry up and satisfaction scores drop. Of course, if you're a super sweet empathetic person but your clinic runs hours behind, you may find a different set of problems! It's all about finding a balance and this really gets at the heart of the "art" of practicing medicine.
One of our attendings is arguably the best example of this I've ever seen. His clinics typically have 50-60 patients per day, always run on time, and he has all his notes done within an hour of the last patient leaving. His patients adore him and laud his bedside manner and personality and he always devotes some time to building rapport and getting to know people on a personal level. Even with resident help, this is a very high volume clinic when you consider that many of these patients get procedures or require some extensive pre-surgical counseling.
So I've made a point to take notes on how her functions because ideally I'd like to emulate that kind of efficiency and empathy when I'm done with training. First, he doesn't typically ask a totally open ended question to begin with, usually alluding to the reason for their referral or their prior surgery or whatever their chronic issue is. He selectively uses open ended questions to elicit key points that really hinge on how a patient describes something, but is quick to focus them in on the key points of information he needs. You can do this politely and patients don't seem to mind since they don't usually want to spend their whole day in the office either. He'll do ROS and some chit chat during his physical exam and then delivers his A&P, answers questions, etc. Then he always ends the encounter by offering to walk the patients out -- I personally and have seen other docs lose time here when patients may want to chit chat, but the walking them out is a very kind and polite way to end the encounter and get them out the door. He then dictates his note immediately which takes him ~1 minute and he's on to the next one.
You'll find it gets much easier to be efficient as you get better at asking questions and knowing exactly what you need to ask for a given issue. Your exams will be faster because you're going for some very key things and either skipping or very quickly blazing through the rest. You'll get better at knowing which patients you have to be more terse with and which you can be more open and still get out the door reasonably fast.
As I'm typing this I remember a gyn attending who taught me more about efficiency in one morning than anything else in med school. I had to have one of those observed and graded encounters, and he did it this way. He told me to go into the room and that if I came out in less than 2 minutes, he'd mark it an honors, 2-3 minutes was a HP, 3-4 a P, and >5 was a fail. He told me the key points he wanted me to get and sent me in --> made it out with 10 seconds to spare and he was true to his word. Even with just 110 seconds, when I presented I realized I had obtained more than enough key information for the visit. This was eye opening to me and he continued to press me for more efficiency throughout the day, always demonstrating it himself as he kindly and personably knocked out visit after visit in just a few minutes, often ending visits with hugs from happy patients and families. Again: empathy and competence are intimately linked and not two disparate concepts that cannot coexist.