Phenols conjugate base is the phenoxide ion. It has four resonance structures. When drawing the resonance structures you'll notice that the negative charge moves from the oxygen to the ortho to the para then to the next ortho position.
Therefore, we observe that the ortho and para position of the phenoxide ion has the most negaitive charactor in these positions.
Now we can answer your question. Attaching a group - wheather it be electron donating or electron withdrawing - to the ortho/para positions will influence the acidity of phenol the most. This is because electron withdrawing groups will stabilize the phenoxide ion's more negative regions best if attached to those sites (spreading out the negative charge if you will). Conversely, electron donating groups will destabilize the phenoxide ion (amplifying the negative charactor).
This leads to the answer - electron donating groups destabilize the phenoxide ion, making it a stronger conjugate base. A stronger conjugate base leads to weaker phenol acidity.