Verbal is just all about sharp focus. Misreading/not understanding a paragraph can throw off your whole section with how critical time is. If there's questions or paragraphs you have to reread multiple times, that makes things more difficult.
Anything that affects your focus will affect your score. There is no correct answer to this. Many people are fine on less sleep. Others are not.