Someone who is your superior at work should never expect a gift of any kind. Something simple is already plenty - a physician, on a physician's salary, has the means to give a more "elaborate and expensive" gift, which they seem to have thought was an appropriate response gift to whatever you gave. I'd bet that they appreciate what you gave them more than you realized.
Also, if this physician allows a holiday gift to significantly affect the rec letter they write, you probably shouldn't be getting a rec from them in the first place