I had a similar conversation with an EM attending a couple years back. I asked him if it would be beneficial to do an outside rotation at a hospital that I am interested in doing residency at. He said "of course." Then I asked him a hypothetical question, "what if I am doing an outside rotation in IM, and I eventually decide to apply at this hospital for Radiology?" He answered, "well, it wouldn't take much effort to walk down to the IM department and ask the PD or attendings what they thought of you."
So even though it is ideal to do an outside rotation in the field you will be applying to, it is not always possible. So doing an outside rotation at that hospital in another field can still be beneficial. Attendings and PDs in the same hospital talk to each other a lot. You can bet they will ask each other for feedback on students that have rotated with them.