For non-rollings, I would say it does NOT matter how early/late you apply. The only non-rolling school I applied to was Duke (primary mid-october, secondary Jan 1, interviewed 2/9, accepted 3/3) and being late didn't hurt me.
That being said, for the love of God, APPLY EARLY!!! For school's with rolling admissions, it really makes a difference. Since I'm sure you'll apply to some non-rollings and some rollings, you should apply to all of them as early as possible. It don't think it's a coincidence that my only acceptance is from a non-rolling school, and I got waitlisted at 3 rolling schools (Pitt, Wake, and Vandy, still waiting to hear from a couple places). Please, don't be like me, learn from people who've been through it and apply as early as possible.