Unfortunately I'm not a cell biologist, so I can only offer my guess after reading Wikipedia and my physiology textbook. Of course, guess ain't ****, but I'll give it anyway.
Possibility 1: the configuration of the channels is dependent on the voltage across the cell membrane. So when the cell is depolarized (i.e. absolute value of voltage is low), the channels are open; when the cell is polarized or hyperpolarized (i.e. absolute value of voltage is ~70-90 or higher), the channels are shut. This could be due to the electronic charges carried by the channel protein and how the charges affect each other.
Possibility 2: the channels open on depolarisation, but can only stay open for a while. So when the cell gets all depolarized like, the calcium channels open and then shut shortly after, no matter the status of the cell. Like think about it like a sponge. You squeeze it and it gets deformed, but you leave it alone after that and it springs back to shape. Maybe the calcium channels can't stay open too long, just long enough for some calcium to influx.
Possibility 3: something else causes it to close. Is it bicarbonate concentration??? Chloride??? Bradykinin??? Magic??? God??? I really couldn't tell ya.