position of authority =/= leadership.
I understand where you're coming from, but if you ever have any doubt whether something is a leadership activity or not, ask yourself if you could answer this question in an interview if it were asked, "You put down refereeing as a leadership position... who did you lead?" You didn't really direct a team in order to accomplish a goal... you were in a position of authority and followed the rules. (totally more difficult than that, I respect what you do! It's just not necessarily leadership)
Now if you worked as an umpire and had a team of other refs working under you, that's being a leader and being the head of a group of your peers and leading them towards a common goal of reffing a good game.
This is just one man's opinion though.