It is...but this happens in an active manner. If the membrane itself was permeable to H+ ions, then the H+ gradient would be dissipated before ATP Synthase could use that gradient to make ATP. It's kind of a 'semantical' question because in other contexts in biology you count transporters when you are considering the 'permeability' of an ion. (Think Na+ conductance when considering resting membrane potential.)
So is the IMM permeable to H+? Well...yes and no. The membrane itself (i.e. the lipid backbone) is not permeable. But H+ can travel from IMS to matrix via ATP synthase, and from matrix to IMS via the ETC (though the ETC H+ translocation mechanism itself is not well understood).