Usually we think of it in terms of basicity. A strong base would not be likely to leave. A weak base would be likely to leave. Think a strong base wants to react complete, so it can't stay alone or leave. If it did leave it would react right away and re-join. Weak one can separate and stay separate.
Example...
In benzene we learned NH2 was the best donating group. -OR is also a good donating group...
In the carboxylic acid derivatives... NH2 is the less reactive compared to esters...
So your theory does not work?