I would also agree to doing work in your field of interest: it's kinda a no brainier.
However, with that said, if you do not have a set field you want to go into or if you are sort of in between a few and something comes up that is a great project, that is a potential field, or even tangentially related I wouldn't disregard it.
For example, I am currently in a lab working on X. I am not 100% sold on this field, although it is a possibility. However, the lab is great, the people mentoring me have good track records and we are doing meaningful/relatively high impact stuff. So as a future physician-scientist, it is a great way to learn the ropes plus it will be fruitful/yield tangible results. I (hope) that as far as your research, even if you are in a totally different field, people (eg. program directors) will appreciate your quality training and good science. This may even be more true if you are working on something super basic science-y and are being judged by clinical faculty later on (and may not be as astute to the basic science aspect of even their own field).
I would also agree however that you (or I in this case) may miss out on a few things: you really do get to know a lot of people in the field (incl. influential/"powerful"), esp if you are working on something with a smaller niche... I am getting to know a lot of folks in this field, but should I switch and do something else later on, knowing them may not be as useful.
Like I said, I am/was in the same boat, but I have sort of come to this conclusion...anyone else would chime in? I hope I'm not wrong!!
Good luck to all the current applicants!