Weren't there rumblings awhile back at how the NIH (or maybe it was another big funding source?) changed the reviewer pool, and the result had been a much lower % of PSYCH/SOCIAL SCI grants being given? I could have sworn I heard one of our research-only faculty lament about some kind of process/review change that really mucked things up.
I hadn't heard anything about this at NIH(and likely would have). These things definitely vary across institute and even committee so, its entirely possible its true for a particular area. Even if true, my grant was heavily pharmacology/neuroscience so I'm somewhat doubtful this would be a major factor. I sent mine through NIDA (though its reviewed through CSR), but mine is dead in line with work that NIDA loves to fund - with Volkow heading it up they have become heavily focused on neurobiology/imaging and similar basic science work aimed at understanding processes involved in addition.
Others - thanks for the kind words. Its just very frustrating...I was prepared for the possibility that it wouldn't do well on the first submission. I was not expecting that to happen on the second submission. Maybe do better but not "good enough" given low paylines, but NEVER thought it would do worse.
As for peer-review, to me I think its just a matter of "least worse" rather than a good system. I've long believed that one cannot tell much from a single manuscript because of the way peer review works. Its a review of the presentation of the science...not the science itself. I can spin things however I want. As I've mentioned in other threads, I'm obsessive with my analyses and if there is one thing I've learned, its that doing so will almost assuredly decrease your confidence in your results. I can take one aim, analyze it 10 different (perfectly legitimate and acceptable) ways and get wildly discrepant results. We can argue we should specify what we will use a priori but this is not always feasible for some designs/circumstances, and it still leaves a whole lot up to chance since there is rarely a single "right" way to analyze data.
I think some of the major problems with grant reviews are:
1) General committees. Many people on these committees will simply not understand enough to evaluate things beyond the broad conceptual level. This can work for or against you of course, but ideally, I think it would be a little closer to manuscripts where decisions are made from a broader pool based on the individual grant. I realize that is somewhat impractical , but I think its important to keep in mind.
2) Reviewer turnover. This is my best guess right now at what cost me. The way the grant review process works, its like every manuscript revision got send back out for review....but to different reviewers. You have no idea what the results will be, and it increases the noise factor. In an ideal world, we would have objective criteria and this would actually be a good thing (more eyes on it, etc.) but in practice, the noise factors seem to outweigh benefits.
3) Reviewer matters. This is true moreso with funding being tight (as I noted previously), but one reviewer who trends towards the middle of the scale may still have rated your grant more highly than any other grant they've reviewed in the last 5 years, but its enough to kill the application in the current funding climate.
4) Related to #2, lack of objective criteria. I recognize this very, very hard to do, but as psychologists (to be) we are arguably the best at doing this. To much is left to the whims of the reviewers. One person may like the idea, another may not. I'm convinced it is all a dice roll at this point. I've seen people get softy reviews from people who probably like everything. Maybe this is what I got on my first submission, maybe I got particularly harsh reviews this time, I don't know. Regardless, the outcomes of a grant submission should not depend as heavily on who the SRO decides to send it to as I wager it does. I'm also curious if
I'm clearly frustrated with the experience, especially since I was relying on this for a career jumpstart given my thesis simply did not pan out. Trying to sort out my options right now in terms of other grants to apply for, how/if this should change my plans regarding internship, post-doc, where I focus my efforts during my remaining time in grad school, etc. For now - I'm sublimating my frustrations by working on manuscripts. Need to make sure they can't turn down the next grant application!
