Yeah been there. I had the same reaction with one of my patients.
I've also had the unpleasant feeling of having failed/disappointed my patients (20 yo can't accept their diagnosis, act out by not taking meds, I try to process this with them - then they've requested another resident - though at least came back and was able to discuss things with me, now we're on the same page).
And for the benzos thing - I have one patient on a benzo out of the 4 or so sites I'm at. Took one guy off a crazy Valium dose (him also being on suboxone). He was signed out to me as "whatever you do, don't take him off his Valium, he'll make your life hell," (ummm, whatever that means). Gave him one chance to do a slow taper, used my script in 10 days and went to another provider (at the same place, and amazingly got another script). I took him down over 3 weeks. He HATED ME. But, he kept coming. After 2 months, he thanked me for getting him off of it, because he realized now that he was abusing it just like he used to abuse alcohol. Now he's doing better without them.
Unless it's very necessary, you're coming to therapy, not missing appointments, reliable, no drug history - none from me