Just from one patient's point of view, it would depend on how 'medical' the scrubs looked. I'd be fine if my Psychiatrist wore something like what Wolfvgang has pictured above (although he's male so I'm going to go ahead and assume perhaps something a little more on the masculine/gender neutral side of the fashion spectrum would be preferable for him), and of course if you can refashion medical looking scrubs into an ultra comfortable non scrub looking outfit, then go for it.
However, if my Psychiatrist wore medical/hospital looking type scrubs at our first meeting, it would have made me extremely uncomfortable unless he had a good, reassuring explanation for why he was wearing them (I associate scrubs on Doctors with potentially needing to rush off at a moment's notice, possibly for a life or limb threatening emergency, so yeah, it would freak me out a bit because I'd be sitting there a) feeling like I was keeping him from something important, and b) expecting him to run out of the room at any second).
If my Psychiatrist randomly turned up in medical scrubs now my first thought would probably be something like, 'You're wearing scrubs? Huh, that's odd', immediately followed by 'Oh god, there's been an Emergency...quick scan for blood stains, did somebody die, was it a patient, did someone get hurt, oh no, is my Doctor okay? Wait, what if his wife's having another baby, and they've had to do an emergency Caeserean, and that's why he's in hospital scrubs, and if he's not at the hospital then something could be really wrong, ****, quick how do I respond to this? Maybe I should stop just standing here looking like a rabbit in the headlights and offer assistance...okay, breathe, just breathe'