I had to back out of an offer 4 days in once, didn't feel good about it, but it had to be done. Wasn't blacklisted (I'm a PGY-1 grad, this was a generalist type position, and part-time only, had to give it up for a FT), in fact they called me a year later trying to recruit me for per-diem (not a hard to staff hospital either).
Had an instance where a PGY-2 specialist backed out for the dream job offer two weeks in, hospital understood but wasn't super happy about it.
Your case though... you're backing out with no other ship to jump to? I don't think you'll be blacklisted or anything but you're taking an awfully huge risk here. At least in the cases above, the pharmacists had security of work in place and as time passed, the downsides to backing out diminished.
Only you know your job prospects best as well as the personality of the department you're dealing with. The hiring team can either understand and move on, or go ape-**** and call all of their colleagues and talk about some snot nosed PGY-2 jilted them at the altar. With no other prospects so far along, I personally would keep the verbal commitment and move forward, do what you can do in 2 years, and move in if it's not for you. Sounds like you accepted pretty quickly, did you not negotiate salary/hourly rate yet? Can you draw it out and fall out on negotiations?