I think I knew somewhere late my intern year or early 2nd year. I had it narrowed down by the time I had decided on IM as a med student - I was deciding between cards, gi, and pulmonary, although even then I was strongly leaning towards pulm.
My program was great in that they let us do a couple of subspecialty electives during the intern year-- mine were GI and ID (i didn't get to do pulm until the beginning of 2nd year). Other required subspecialty inpatient months I had that year were heme/onc, CCU, ICU, and hepatology, so I got a decent exposure to many of the fields I was considering. I was able to rule out GI after doing my 3-week elective in it early in the year. Then I did the CCU/inpatient cardiology service -- liked the rotation, but I softly ruled out cards after doing that month, although I kept the option open in case I decided I didn't like pulmonary. I then did ICU and pulmonary consults almost back-to-back (ICU at end of internship and pulm elective in the beginning of 2nd year). Luckily I really liked both of them, and solidified my subspecialty decision by August of 2nd year.
I think it helps to have narrowed down your list of potential subspecialties somewhat before you start residency, however you should still approach the subspecialties with an open mind. I know many people who changed their minds during residency. On the other hand, if you go in wanting to try out all the subspecialties, you won't make your decision until the end of your last year, especially since most of your elective time isn't until 3rd year. There's nothing wrong with deciding late, though -- lots of people do wait to decide, and they will take 1-2 years off before fellowship to work and make money.