My GPA got rolled last semester due to my interviews. Depending on how many you go on and where at (e.g., driving vs. flying), performance in class will inevitably drop. I missed about 12 class days for interviews, which comes out to almost two and a half weeks. It's hard to make up that time IMO.
That said, bring your work with you to interviews and try and get as much done as possible. But make sure you enjoy the trips and explore the cities/schools as much as you can. It's not very often that you have a good reason to travel across the country and see a whole bunch of new places.