I don't think that you understand what "equal pay for equal work" means. Until the 1970's, it was perfectly legal to pay women less per hour for doing the exact same job that men did. So, for example, a school could pay a male 4th grade math teacher $20,000 a year, but might pay a woman doing the exact same job $12,000 a year. The justification might be that the man has to support a wife and children, but a married woman doesn't need that much money since her husband is the primary breadwinner, while a single woman only has to support herself. So if you "don't buy the whole equal pay for equal work thing" then you support paying women less for the same job.
Unfortunately, in some industries, where pay is not standardized, such pay discrepancies persist.
If what you meant to say was that you feel that statistics showing that women are paid less than men are misleading due to women being more likely to work part time or take off for several years to raise children, then I might agree with you that some, but not all, of that difference might be due to voluntary reduction of time spent at work.
