Have you tried communicating this assertively and directly to your supervisor? You're a doctor now... I think you can push your weight around a little bit more than you probably could have on internship and practicum; e.g., stating that you're going to do this thing that everyone seems to be on board with vs. asking if you can do this thing that everyone seems to be on board with.
I'm imagining a conversation, like: "hey, I'm finding myself with extra time on my hands, so I'm going I'm going to start seeing x number of patients with x supervisor for the next four months. This will help me round out my training in x, y, and z ways. Over the next four months, can you and I talk about ways to increase my caseload on my primary rotation with you?"
Start sending emails to set up supervision times with whomever would oversee this mini rotation, *maybe* (maybe...) cc your TD on emails with your primary supervisor if you anticipate pushback *and* think/know your TD is on board with you seeing some patients with a supplemental supervisor, etc.
Maybe also schedule a meeting with your TD and primary supervisor. You're still a trainee, but there is an onus on you to be assertive and proactive about reaching your goals for fellowship.
I echo the majority of the other folks on here that it seems highly unlikely that leaving your postdoc prematurely would yield a positive result. Heck, finishing this fellowship and completing another 1-year postdoc somewhere else would be preferable to burning a bridge by leaving early.