Think of the difference between exudate and transudate. If the protein content of the subcutaneous fluid is low, pressing on the edematous area will cause a depression ('pitting'). Vice versa, edema in diseases like acute glomerulonephritis will be hard ('non-pitting').
This is evident in location of edema as well. Pitting edema is evident in lower extremities, especially ankles. On the other hand, edema of eyelids is seen in glomerulonephritis.
Maybe one point to make here is about thyroid diseases. In hypothyroidism and Graves' disease, connective tissue elements (esp. GAG) are deposited in subQ tissue, resulting in myxedema - which is a non-pitting edema (again, this is due to the nature of the deposit).