First, because there often isn't a context where the word has the correct meaning in the first place. If you have a list of disallowed curse word substrings and Dr. Theodore Fucknuckett gets upset at not being allowed to login, then adding "Fucknuckett" to something called a "whitelist" is using a somewhat tenuous connection to the meaning of whitelist. "Allowlist" is more direct. Also, I just made it up, and I gotta say that Dr. Ted has an awesome last name that I'm going to have to reuse for something sometime.
Dictionary meaning of whitelist:
1. A list of people or organizations that have been approved to receive special considerations or privileges.
2. A list or collection of people or entities that are known, trusted, or explicitly permitted.
Are you giving the word "Fucknuckett" special privileges? Sorta. Is Dr. Fucknuckett explicity permitted? Kinda. His name is, at least.
Master vs slave is different but similar. They are often used in a somewhat strained metaphorical way. There are often better terms. Client/server, primary/secondary, primary/replicant, authoritative/copy. (I don't actually have a problem with "master copy".)
Second, often the only context from which you are deriving meaning is the slavery meaning of master/slave. A master/slave database setup gets its meaning from the master telling the slaves what to do. And once again, it's a strained metaphor, since the significance of the master/slave setup is usually about the data, not the control.
Anyway, the point is that other definitions of the word "master" are irrelevant if you're drawing upon the slavery meaning in the first place.
Those don't invalidate your argument, but if you're balancing the usefulness of terms against potential offense or whatever—well, I argue that the usefulness is not all that high.
Third, relying on the cultural meaning of colors is problematic for the very reasons you cite. I'll give another: Lucifer means "light-bringer". How is anyone supposed to guess which meaning you're drawing upon for a given context unless they already know?
I also happen to be sympathetic to the argument that introducing more negative associations to the word "black" is a problem, given the large number that are already in use. But I'm not arguing that here; I'm just arguing that these words and contexts aren't useful in the first place. And that when the context *is useful, it's very often drawing upon the slavery meaning. (Whether that's a reason to stop using it is a separate question.)