ASCP and ABP have looked into this relationship and are currently working on getting that info published. That is all I know about it right now. So be on the lookout for the article (they did not mention which journal it might be in).
I'm gonna guess it's going to be a while. The score for each section goes into the 6-700 range, right? Look at the difference between PGY1-4 for each section, you'll see it's only like 10-20 pts/year (that's like a 2% difference in score!!!). The standard deviation within each class is likely significantly more than the difference between classes. To see any statistical difference, you'd probably need like 100K people taking this test. Also, you can even account for the rising score on the fact that there are repeat questions, giving more senior residents an advantage in score. And the statistical difference you do see will probably be no better than something like: "scoring a 500 significantly increases your chance of passing the boards by 0.25%". Furthermore, this test itself likely means nothing for board prep- as people who are good at taking tests, well, do well on tests. If you did well on your MCAT, STEP1-3, etc, you will probably do well on the boards regardless of what your RISE score is (although you probably did well).
I think the problem lies in the fact that:
1. People take this test for a variety of different reasons, and there is no "passing" grade. So some people will study, while others do not. If a 2nd year studies a section very hard, they are likely to do as well or better than a 4th year who does not prepare.
2. Monitoring. There is none, formally. Some places this is a group test, destroying any meaning to a cumulative score. Some people take it at home with their books. Some take it "proctored" with open books. Who knows what goes on?
3. Some programs have formal tutoring/test prep sessions. These likely include questions remembered from previous years. Other programs do nothing like this. I bet if you look at the scores in the former programs they will be significantly better than the latter.
4. Some programs use your score to punish you. Those people will be motivated to study and do well. Others have no motivation to even try.
Bottom line: I think this test is a complete waste of time and money. The only possible use is to help guage your own progress, assuming you approach the test the same way every year.