Yes you will be at a temporary disadvantage.
Every city has the following.
The idiot psychiatrist that gives every patient what they want even if it's hurting the patient such as Xanax, the one that diagnoses everyone with bipolar disorder, the one that is into some type of bogus holistic treatment (e.g. ginseng for schizophrenia), one that only spends 5 minutes per patient and is rude to them.....
By the time you finish residency you have a decent idea of who the good doctors are the vs the bad ones, the good gigs vs the bad ones, what the going rate of pay is, what to expect in terms of how easy it will be to make a private practice.
These things are location specific. You will need to have a lot of good local knowledge to figure this stuff out.
By the time I left Cincinnati my patients showed me a list of which doctor they should take to replace me. I looked at the list and was able to check off 1-avoid, 2-avoid, 3-avoid, 4-decent, 5-okay that one's good, 6-avoid, know what I exactly wanted to do had I stayed (too bad I was leaving) and entered a new place where it took me about 6 months to realize I wasn't going to like my job, another 6 months to figure out what was the best exit strategy (and I really had to actively network on that one) and then eventually get out of it.
Been in the new location for 1.75 years now and I still don't know anywhere near of what I knew of Cincinnati in terms of psych opportunities. By the time I left Cincinnati almost every mental health provider knew who I was and I established a good rep in that city.