I would think for those who are married with kids and looking to either go home or find a place to settle down, having more variety in residencies to match into would be better.
If you know you want to go back to SoCal in the end. Why not match there and get the ball rolling? A good USMLE will only increase the number of locations for you.
However, if your dodging it because you can't do well, that's something that should be addressed.
I think you're right about about people regretting not taking it much of the time, but hindsight is 20/20.