The way I understand it, imagine if you were in space and looked at the sun, it would just be the sun surrounded by black. The only light you would see is the yellow light coming straight from the sun, light wouldn't be refracted in any other direction, so everything else would be black.
If anything most of the sky would be yellow given the Sun's size.