That's the whole fallacy. Yes, it matters. Your choices and action do matter. Naturally death can strike at any time and for any reason, or simply no reason at all. However, why tempt fate? Why would you want to facilitate death? I mean it's not refutable that smoking hastens death. If you continue smoking with that knowledge, you are having a sort of death wish, then, aren't you?
I'm not saying you shouldn't enjoy your life. Just make your choices consciously, knowing the facts and consequences of your actions. If you choose to smoke, then so be it. Just know that it can hasten death or prolong your suffering (which can be worse than death, in my opinion...I mean, have you seen someone writhing in agony from oxygen starvation in end-stage disease?).