According to the ASCP website, "RISE results should not be used as the sole criterion on which performance, promotion, or advancement is based." A program which is essentially firing residents based on their RISE scores is, I think, putting themselves at risk. It's also been commonly said that some programs allow their residents to take the RISE open-book or by pooling knowledge/talking, as much a learning experience as a test, which should sway the overall averages.
Personally, I think of the RISE as a reasonable way to gauge your own progress over the years vs yourself and vs your peers, on average, and a reasonable facsimile of what the boards can be like -- different but the same. I wouldn't put much weight on 1st year results, especially in areas you haven't had a rotation in yet. But you certainly want to see significant improvement year to year, and by your final year it's at least some sort of standardized gauge to assess your readiness for the boards. Of course, it can also be misleading, or you inadvertently find yourself not studying something for the boards because you did so well on the RISE and end up not passing the boards. But, it's a heckuva lot better than nothing.