Thanks! When they said yes,do you still send them periodic reminders after you started to apply or before you started to apply?
Just give them a deadline that's earlier than when you REALLY want to have them done by. Whether that's before you send in your primary or after, is up to you. This way, like jared_the_great said above, it's not the end of the world if they miss the deadline you told them. However, if they pass that deadline you told them, it would be a good idea to remind them about your LOR.
And how to send them periodic reminders? I wonder if I send them too much, will it be bothering them or annoying them?
I would use your judgement based on how comfortable your LOR writer is with you. I wouldn't recommend more than once a week in most cases. My LOR writers knew me well enough, so neither I nor they really found the reminders to be an annoyance. They're busy people, and they acknowledged that reminders were a good idea otherwise because otherwise they'll lose track of time and miss the deadline.
I have a feeling that "straggler" LOR won't be good anyway.
Good LOR usually won't need any reminder. The writer will probably finish it ahead of time and send to you for your opinion, then he/she upload it into the system.
Therefore I really hate to ask any "straggler" LOR writer... It is annoying to them, and me too.
One of my letters was from a physician who I work very closely with on a regular basis. He actually requested on his own that I remind him about my letter and "bug" him often because otherwise he'd forget. He has a million responsibilities on his mind, after all. He got my letter in before the deadline I told him.
The other one I reminded was a professor who, again, knows me well since I talk to her often and have TAed for more than one class. She's the only one who went past the deadline I told her. It took two reminders for her to get it in, that's all.
I sent reminders to both these people, and both were kind enough to give me copies of the letters (even though I had not asked to see and I had waived my right). I read them, and I couldn't have asked for stronger, more thoughtful, and personalized letters. I'm incredibly grateful for them.
How good the letter is and how much of an "annoyance" reminders are--stragglers or not--just depends who you're asking and your relationship with them.