But this is exactly why governments need to exist to protect the rights of citizens. Like you said, we had actual slavery in this country, and slavery still exists in a myriad of forms; do you honestly think that slaves would have been better off had they simply "actively worked and trained to arm and protect themselves"? People who are enslaved and exploited are by definition less powerful than those who exploit them, and no manner of self-armament would have protected those slaves from becoming enslaved. Moreover, it's not ethical from a moral standpoint to expect all groups to be able to protect themselves from this form of subjugation, and having this expectation is dangerous because it could lead one to justify subjugation by saying "well, they should have been able to protect themselves".
Slavery continued to exist in America because the government allowed it to however, once the government began to outlaw it, that's when it finally broke down. If there was no pressure from the government, there would have been no financial incentive for slave-owners to stop owning slaves, and it would have remained in practice until it was no longer financially beneficial.