Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Decision trees – the unreasonable power of nested decision rules

mlu-explain.github.io

Post reply on HN
Print view View original

71–80 of 89 posts

Re: Decision trees – the unreasonable power of nested decision rules

#71

Post by getpokedagain »

jjcc wrote: post #39
getpokedagain wrote: post #35

I worked (professionally) on a product a few years ago based upon decision tree and random forest classifiers. I had no background in the math and had to learn this stuff which has payed dividends as llms and AI have become hyped. This is one of the best explanations I've seen and has me super nostalgic for that project. Gonna try to cook up something personal. It's amazing how people are now using regression models…

I worked on a product which was the best ID reader in the world at the time 25 years ago. The OCR engine was based on Decision tree and "Random Forest" (I suspect the name did exist) with only 3 trees. It was very effective as a secret weapon of the competitiveness. I tried to train a NN with a framework called SNNS(Stuttgart Neural Network Simulator) as the 4th tree complement to the existing 3. Today, hand writing…

That's awesome and based on my experience I'm not shocked this went well. I'm not sure what the features would be in this but I am assuming they could be specific pixel combinations or other things which would be easily labeled in a few ways. I hope you had fun with it.

My previous project was far from that. https://healthverity.com/audience-manager/

I had a lot of fun, really the last fun project I've had. I hope you had fun as well.

Re: Decision trees – the unreasonable power of nested decision rules

#74

Post by fooker »

cubefox wrote: post #65
fooker wrote: post #53

Earlier quoted context omitted.

> I still don't know exactly what you mean Straight forward quantization, just to one bit instead of 8 or 16 or 32. Training a one bit neural network from scratch is apparently an unsolved problem though. > The trees that correspond to the neural networks are huge. Yes, if the task is inherently 'fuzzy'. Many neural networks are effectively large decision trees in disguise and those are the ones which have potential…

> Training a one bit neural network from scratch is apparently an unsolved problem though. It was until recently, but there is a new method which trains them directly without any floating point math, using "Boolean variation" instead of Newton/Leibniz differentiation: https://proceedings.neurips.cc/paper_files/paper/2024/hash/7...

Nice!

Re: Decision trees – the unreasonable power of nested decision rules

#75

Post by fooker »

fc417fc802 wrote: post #64
fooker wrote: post #53

Earlier quoted context omitted.

> I still don't know exactly what you mean Straight forward quantization, just to one bit instead of 8 or 16 or 32. Training a one bit neural network from scratch is apparently an unsolved problem though. > The trees that correspond to the neural networks are huge. Yes, if the task is inherently 'fuzzy'. Many neural networks are effectively large decision trees in disguise and those are the ones which have potential…

> Training a one bit neural network from scratch is apparently an unsolved problem though. I don't think it's correct to call it unsolved. The established methods are much less efficient than those for "regular" neural nets but they do exist. Also note that the usual approach when going binary is to make the units stochastic. https://en.wikipedia.org/wiki/Boltzmann_machine#Deep_Boltzma...

Interesting.

By unsolved I guess I meant: this looks like it should be easy and efficient but we don't know how to do it yet.

Usually this means we are missing some important science in the classification/complexity of problems. I don't know what it could be.

Re: Decision trees – the unreasonable power of nested decision rules

#76

Post by fc417fc802 »

fooker wrote: post #75
fc417fc802 wrote: post #64

Earlier quoted context omitted.

> Training a one bit neural network from scratch is apparently an unsolved problem though. I don't think it's correct to call it unsolved. The established methods are much less efficient than those for "regular" neural nets but they do exist. Also note that the usual approach when going binary is to make the units stochastic. https://en.wikipedia.org/wiki/Boltzmann_machine#Deep_Boltzma...

Interesting. By unsolved I guess I meant: this looks like it should be easy and efficient but we don't know how to do it yet. Usually this means we are missing some important science in the classification/complexity of problems. I don't know what it could be.

Perhaps. It's also possible that the approach simply precludes the use of the best tool for the job. Backprop is quite powerful and it just doesn't work in the face of heavy quantization.

Whereas if you're already using evolution strategies or a genetic algorithm or similar then I don't expect changing the bit width (or pretty much anything else) to make any difference to the overall training efficiency (which is presumably already abysmal outside of a few specific domains such as RL applied to a sufficiently ambiguous continuous control problem).

Re: Decision trees – the unreasonable power of nested decision rules

#77

Post by levocardia »

One of the ML textbooks (ESL maybe?) I read described decision trees as (paraphrasing) "really great - they are interpretable, fast to fit, work on lots of different types of data and outcomes, insensitive to scaling and distributional issues, don't have too many tuning parameters...except they just don't work very well." That latter problem can be solved with bagging or boosting, though you are bargaining away many of the other advantages.

Re: Decision trees – the unreasonable power of nested decision rules

#78

Post by levocardia »

jebarker wrote: post #49

The killer feature of DTs is how fast they can be. I worked very hard on a project to try and replace DT based classifiers with small NNs in a low latency application. NNs could achieve non-trivial gains in classification accuracy but remained two orders of magnitude higher latency at inference time.

Also, decision trees (but not their boosted or bagged variants) are easy (well, easy-ish) to port manually to an edge device that needs to run inference. Small vanilla NNs are as well, but many other popular "classical" ML algorithms are not.

Re: Decision trees – the unreasonable power of nested decision rules

#80

Post by jjcc »

mistrial9 wrote: post #57
jjcc wrote: post #39

Earlier quoted context omitted.

I worked on a product which was the best ID reader in the world at the time 25 years ago. The OCR engine was based on Decision tree and "Random Forest" (I suspect the name did exist) with only 3 trees. It was very effective as a secret weapon of the competitiveness. I tried to train a NN with a framework called SNNS(Stuttgart Neural Network Simulator) as the 4th tree complement to the existing 3. Today, hand writing…

in the interest of understanding, is there any code or similar for the approach? does that OCR run anywhere today?

The technology was developed by my predecessor during late 90s when microprocessors was much less powerful, and the resolution of image sensor was low. The relatively high accuracy based on those conditions was a critical factor to use Decision Tree as OCR engine. It's used till 2007 when I left my company.

I don't think it would survive afterwards due to quick change in technology. Even the desktop OCR applications at the time didn't use Decision Tree because the CPU was much more powerful. The DT OCR engine was competitive only under special use case.

Post reply on HN