This is how I reasoned it out:
Cl- is a moderate/poor nucleophile. But the protonated hydroxy group is a very good leaving group.
With tert alcohols, the step when the L.G. leaves (1st step) is the rate determining step via SN1. In this case, the RTS happens very fast because the LG is so good and because the carbocation is tertiary.
With primary alcohols, it can only leave via SN2. But how well the L.G. is doesn't matter as much because it all happens in 1 step. And the nucleophile isn't that great (halogens are only moderate to weak nucleophiles). So the reaction is slower.
To summarize, let's look at the reaction rate laws.
SN1 (tert): rate = k[alcohol]
SN2 (primary): rate = k[alcohol][nucleophile]
The first reaction will go fast because it only depends on how good the alcohol is. The second reaction will not go as fast because it depends on how good both the alcohol and the nucleophile is.
This might not be the correct way to think about it.