I have not applied, though I intend to in a year or two. From what I have heard it is quite competitive, though not unreasonably so. The main difficulty stems from the fact that due to the NIH timeline, most people NEED them funded on the first try. Most NIH grants you get 3 submissions for, but going through the full process 3 times will take quite some time (I'd guess about 2 years). Few people are willing to delay graduation that long in order to get one funded. I hope I'm not in that position, but if I am I would actually consider doing so if I'm reasonably on-track prior to that point. The way I see it, the sooner I can get my hand in the NIH cookie jar, the better off I will be.
People typically apply for the F31 when preparing their dissertation. When you apply would depend on what your program's timeline is like. As far as how developed it needs to be...its a grant. It basically needs to be your dissertation proposal, just in grant format (in other words, no 50 page introductions
). People used to submit them early on, long before having a firm dissertation project, and get funding for all of grad school...to the best of my knowledge that does not happen anymore, and you need a clear project to even be considered.
It is not only the norm for your POI to assist, but a requirement. The grant needs a faculty sponsor, which will generally be your advisor, and often a few other individuals (consultants, etc.). Many people write in training elements to the grant, to pay for workshops, etc. Again, in order to be competitive, they typically need to be seasoned investigators, and at least some need to have obtained NIH grants of their own. It isn't necessary, and I know someone who got one despite having an advisor who never had even a small NIH grant, but its surely helpful.
I know we have some folks here who have been through the process, so I'm sure they can give you more insight into what the actual process is like, and correct me if I'm off base on anything