This discussion is getting close. Let's review. In the general acyl substitution mechanism a nucleophile attaches to the trigonal planar carbonyl carbon resulting in a tetrahedral intermediate. The tetrahedral intermediate then resolves itself in the next step with the departure of the leaving group. In the case of acyl substitution of a carboxylic acid, the leaving group is hydroxyl group, while subsequent to the nucleophilic attack on an acyl chloride or bromide, the leaving group is the halide anion.
Why are acyl halides the most reactive?
Whenever the question is about reactivity, firstly it's important to understand whether the context is about kinetics or thermodynamics, because reactivity is a word, like stability, which is used for either case. In the context of acyl substitution, the context is about thermodynamics. In other words, acyl halides are the most reactive, especially compared to carboxylic acids because they are disfavored by equilibrium.
Remember that the position of equilibrium in a reaction is determined by the standard free energy change. The free energy change can be assessed by thinking of the change in enthalpy (or heat flow) and the change in entropy. Although there are differences in the entropy function because of the different degrees of order the ions, hydroxide versus chloride or bromide, impose on the water molecules in the solvation process, the most decisive consideration is going to be the enthalpy function.
Because in this case you are talking about a constant volume transformation, you can substitute thinking about internal energy change for enthalpy change. Basically, then, why is a chloride anion in solution more stable, i.e. lower energy, than a hydroxide anion? It's a question of electrostatic potential energy. The gist of it is that chloride and bromide are much bigger anions. Think about it in terms of capacitance. Chloride anions and bromide anions, being bigger, have a greater capacitance. If they were metal spheres in physics we would say that the bigger sphere holds the same amount of charge at lower voltage.
So a process in which a chloride anion or bromide anion is formed is generally going to be thermodynamically favored over an analogous process where a hydroxide anion is formed. This is why acyl halides are more reactive in acyl substitution than carboxylic acids, and also why hydrochloric acid and hydrobromic acid are stronger acids than water.
However, acyl fluorides aren't all that reactive. Fluorine is smaller, so there is higher potential energy in trying to stuff that charge onto a small sphere. Fluoride isn't as good a leaving group, and hydrofluoric acid is a weak acid, unlike hydrochloric and hydrobromic. The primary reason is because of electrostatic potential energy in how spread out the negative charge is from the other electron density on the anions which underlies comparative differences in internal energy going toward differences in enthalpy and ultimately the standard free energy change which determines the equilibrium constant. I hope this makes sense!
