Earlier quoted context omitted.
I'm keeping white/black and greytones out for now. White/light greys are especially hard... how would you differentiate between the content and the background? It's definitely on the list of things to address.
Any chance of open sourcing? I'm interested in the algorithm. Is it quantizing to a more limited bit depth and taking say, top 5ish via mode?
At the moment it's quantizing down to 40 colours with a tree depth of 4, I then pull the top 12.