It depends on the hospital, but generally the average pharmacist isn't going to have a lot of patient interaction, so I don't think that would be the issue with switching from managed care to hospital. Some hospitals will have pharmacists doing prescription discharge counseling, but a lot of hospitals still have nurses doing that. Hospital pharmacists might be asked to do outpatient talks or brown-bag session, but those are more occasionally, and not a daily part of the job.
The bigger issue would be how comfortable are you with IV's and other drugs primarily found in hospitals? If you have worked with these as a clinical pharmacist, then that would be a great selling point to a hospital. Some hospitals might screen for residency and automatically disqualify you, but if you are applying to smaller hospitals, especially outside of big cities, I think you could find a hospital to take you.
Is there a reason why you want to change your job? It sounds like you have a nice unicorn job and are doing a good job at it, why would you want to risk changing to a different job?