Yes it is pass/fail with quartiles. That being said, I talked to the dean about this and they said that, like anywhere you go, your Step 1 is what really matters. This is for 2 reasons 1. a majority of schools don't rank so PDs can't compare students against a null value. Therefore, most program directors don't up put much stock in it. 2. a person might be 1st in their class at a smaller, less competitive school but would be last in their class if they went to a top tier school. So it doesn't offer the best comparison.
EDIT: this doesn't usually hold with the ROAD specialties (Radiology, Ortho/Optho, Anesthesiology, Derm) and various others these days (PMR and EM) but to that point, see the next answer.
As a current student, I definitely struggled with this at first because I came from a college where grades were very harsh, but we weren't ranked against each other until graduation (cum laude, magna cum laude etc.) But, as of late, I've seen how much the soft ranking has pushed and motivated to learn the material better, and more in depth, and this motivation will *hopefully* contribute to a higher step score. That being said, I definitely think that this contributes to Cinci's SUPER HIGH step score. so yah, it might not be the most comfortable in the short term, but it seems to pay off significantly in the end.