I wouldn't worry about it.
The "Chairs letter" is a misnomer. If done well, it has nothing to do with the chair. It is a mini-dean's letter that talks only about your IM experiences in detail. So it usually has your grades in your clerkship, plus any electives / SubI / 4th year medicine rotations, along with some sense of how well you've done compared with everyone else going into medicine. So whether you worked with the chair or not is meaningless, it's usually written by the clerkship director, or by a small group that writes all the letters and standardizes them. The useless/bad chair's letters are the ones actually written by the chair that say "I've never worked with this person, but they seem kinda nice".