Few things here
1. Not sure if you worked in pain management, but "pain management hours" are not as cushy as they sound. Most jobs average 4 weeks vacation. If you are a private practice owner, I imagine it would be tough to allow yourself to take more than 4-6 weeks vacation due to lost income and patients being upset. On any given day when you leave work, you will be hunted down if you forget to refill meds, call back a patient, or some other task. Not even including finishing notes here or other administrative tasks which is another layer of taking work home. Seeing an obscene amount of pain patients 4-5 days a week even if its just for 8 hours in a day will burn you out real quick unless the big $$$ are rolling in.
2. Regarding "things can go south immediately" in bread and butter anesthesia, presumably this is why you did a residency. You are trained well to prevent such scenarios and hence why most day to day anesthesia you dont have patients dying in every case even if they are ASA 3 and 4. I would guess majority of patients in a hospital anesthesia setting average ASA 2-3 so this shouldn't be something "scary" to the average anesthesiologist. Anesthesia for sure has its downsides, but bread and butter cases routinely going south is not realistic for most providers (MD and CRNA).
Not sure why you are trying to scare OP from going back to anesthesia. Do you feel super stressed in every single case you do?
For context: Pain trained, practiced few years and left to do locum anesthesia for all the reasons OP mentioned.