Yes it matters. Does it matter alot? Given the data now, it does not.
First, QR affects the academic average. In addition, you do not want any section score glaringly low - even QR.
Considering its not hard to get an average score in math (around 17) I would study for it.
Don't get the impression that one can not study math and get like a 13, and expect to be alright!
The moral is to study hard for everything. The fact is the exam assigns you a score for the subjects tested - math included. The admissions committees compares each applicant's scores. Given the situation where all other things to consider are equal among two people, wouldn't you rather be the one with the higher QR score?
Something to think about.