It's definitely doable and common nowadays. In terms of how "bad" it will be, it really comes down to the "why" you are transferring. If you are transferring into a better program or have legitimate family issues, this is typically well accepted. If you are transferring due to poor performance, this can be problematic cause, the 1st resident director can be involved in all your future credentialing.
As states above, the 1st director does have the power to control whether you can leave. They must release you in order for you to transfer and they will talk to any program you wish to join.