Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Decision trees – the unreasonable power of nested decision rules

mlu-explain.github.io

Post reply on HN
Print view View original

81–89 of 89 posts

Re: Decision trees – the unreasonable power of nested decision rules

#81

Post by vintermann »

jjcc wrote: post #39
getpokedagain wrote: post #35

I worked (professionally) on a product a few years ago based upon decision tree and random forest classifiers. I had no background in the math and had to learn this stuff which has payed dividends as llms and AI have become hyped. This is one of the best explanations I've seen and has me super nostalgic for that project. Gonna try to cook up something personal. It's amazing how people are now using regression models…

I worked on a product which was the best ID reader in the world at the time 25 years ago. The OCR engine was based on Decision tree and "Random Forest" (I suspect the name did exist) with only 3 trees. It was very effective as a secret weapon of the competitiveness. I tried to train a NN with a framework called SNNS(Stuttgart Neural Network Simulator) as the 4th tree complement to the existing 3. Today, hand writing…

And I still can't find a big NN model which reads historical handwriting well.

Re: Decision trees – the unreasonable power of nested decision rules

#82

Post by imtringued »

fooker wrote: post #53
smokel wrote: post #25

Earlier quoted context omitted.

> single bit neural networks are decision trees. I didn't exactly understood what was meant here, so I went out and read a little. There is an interesting paper called "Neural Networks are Decision Trees" [1]. Thing is, this does not imply a nice mapping of neural networks onto decision trees. The trees that correspond to the neural networks are huge . And I get the idea that the paper is stretching the concept of de…

> I still don't know exactly what you mean Straight forward quantization, just to one bit instead of 8 or 16 or 32. Training a one bit neural network from scratch is apparently an unsolved problem though. > The trees that correspond to the neural networks are huge. Yes, if the task is inherently 'fuzzy'. Many neural networks are effectively large decision trees in disguise and those are the ones which have potential…

>Many neural networks are effectively large decision trees in disguise and those are the ones which have potential with this kind of approach.

I don't see how that is true. Decision trees look at one parameter at a time and potentially split to multiple branches (aka more than 2 branches are possible). Single input -> discrete multi valued output.

Neural networks do the exact opposite. A neural network neuron takes multiple inputs and calculates a weighted sum, which is then fed into an activation function. That activation function produces a scalar value where low values mean inactive and high values mean active. Multiple inputs -> continuous binary output.

Quantization doesn't change anything about this. If you have a 1 bit parameter, that parameter doesn't perform any splitting, it merely decides whether a given parameter is used in the weighted sum or not. The weighted sum would still be performed with 16 bit or 8 bit activations.

I'm honestly tired of these terrible analogies that don't explain anything.

Re: Decision trees – the unreasonable power of nested decision rules

#83

Post by fooker »

imtringued wrote: post #82
fooker wrote: post #53

Earlier quoted context omitted.

> I still don't know exactly what you mean Straight forward quantization, just to one bit instead of 8 or 16 or 32. Training a one bit neural network from scratch is apparently an unsolved problem though. > The trees that correspond to the neural networks are huge. Yes, if the task is inherently 'fuzzy'. Many neural networks are effectively large decision trees in disguise and those are the ones which have potential…

>Many neural networks are effectively large decision trees in disguise and those are the ones which have potential with this kind of approach. I don't see how that is true. Decision trees look at one parameter at a time and potentially split to multiple branches (aka more than 2 branches are possible). Single input -> discrete multi valued output. Neural networks do the exact opposite. A neural network neuron takes m…

> I'm honestly tired of these terrible analogies that don't explain anything.

Well, step one should be trying to understand something instead of complaining :)

> Single input -> discrete multi valued output.

A single node in a decision tree is single input. The decision tree as a whole is not. Suppose you have a 28x28 image, each 'pixel' being eight bits wide. Your decision tree can query 28x28x8 possible inputs as a whole.

> A neural network neuron takes multiple inputs and calculates a weighted sum, which is then fed into an activation function.

Do not confuse the 'how' with 'what'.

You can train a neural network that, for example, tells you if the 28x28 image is darker at the top or darker at the bottom or has a dark band in the middle.

Can you think of a way to do this with a decision tree with reasonable accuracy?

Re: Decision trees – the unreasonable power of nested decision rules

#84

Post by Otterly99 »

lokimedes wrote: post #52
ekjhgkejhgk wrote: post #48

Earlier quoted context omitted.

I used to be in physics but theory, not experiment. I have experience at work with decision trees in a different field. I've always thought that the idea that decision trees are "explainable" is very overstated. The moment that you go past a couple of levels in depth, it becomes an un-interpretable jungle. I've actually done the exercise of inspecting how a 15-depth decision trees makes decision, and I found it impos…

I completely agree, as you may infer from my comment. The second multivariate models are relevant we effectively trade explainability for discrimination power. If your decision tree/model needs to be large enough to warrant SGD or similar optimization techniques, it is pretty much a fantasy to ever analyze it formally. My second job after physics was AI for defense, and boy is the dream of explainable AI alive there.…

Do you have any good resource on explainable AI (XAI)? Most books I read on the subject tend to be a bit shallow and not very informative.

Re: Decision trees – the unreasonable power of nested decision rules

#85

Post by YeGoblynQueenne »

lokimedes wrote: post #27
srean wrote: post #23

Earlier quoted context omitted.

> Times have changed… This makes me a little concerned -- the use of parameters rich opaque models in Physics. Ptolemaic system achieved a far better fit of planetary motion (over the Copernican system) because his was a universal approximator. Epicyclic system is a form of Fourier analysis and hence can fit any smooth periodic motion. But the epicycles were not the right thing to use to work out the causal mechanics…

If you sum up experimental physics into one heuristic it is “avoid fooling yourself with assumptions” - I left physics over a decade ago, but I feel confident that physicists still work hard to understand what they observe and don’t let LLMs have all the fun. If there’s one field of science where the scientists are legitimately allowed to go all the way back to basics, it’s elementary particle physics.

This guy says physicists now leave most of the work to LLMs:

https://www.youtube.com/watch?v=PctlBxRh0p4

(Original title was "Physicists are surrendering to AI" but it was changed since to something less clickbaity, which I appreciate. I posted it here on the original title: https://news.ycombinator.com/item?id=46859804)

Re: Decision trees – the unreasonable power of nested decision rules

#86

Post by thesz »

3abiton wrote: post #45
srean wrote: post #19

A 'secret weapon' that has served me very well for learning classifiers is to first learn a good linear classifier. I am almost hesitant to give this away (kidding). Use the non-thresholded version of that linear classifier output as one additional feature-dimension over which you learn a decision tree. Then wrap this whole thing up as a system of boosted trees (that is, with more short trees added if needed). One of…

I think it's worth mentioning, that the achilles heel of DT, is in fact, data (more specifically feature) engineering. If one does not spend significant time cleaning and engineering the features, the results would be much worse than, say a "black box" model, like NN. This is the catch. Ironically, NN can detect such latent features, but very difficult to interpret why.

  > Ironically, NN can detect such latent features
Like they detect magazine logo on images of horses?

Re: Decision trees – the unreasonable power of nested decision rules

#87

Post by thesz »

srean wrote: post #68
datsci_est_2015 wrote: post #66

Earlier quoted context omitted.

This varies so wildly from domain to domain. Highly structured data (time series, photos, audio, etc.) typically has a metric boatload of feature extraction methodology. Neural networks often draw on and exploit that structure (i.e. convolutions). You could even get some pretty good results on manually-extracted neural-network-esque features handed off to a random forest. This heuristic begins to fall off with deep l…

Ah! Your comment helped me understand the parent comment so much more. I thought it was more about data hygiene needs. Yes a DT on raw pixel values, or a DT on raw time values will in general be quite terrible. That said the convolutional structure is hard coded in those neural nets, only the weights are learned. It is not that the network discovered on its own that convolutions are a good idea. So NNs too really (da…

  > So NNs too really (damn autocorrect, it's rely, rely) on human insight and structuring upon which they can then build over.
If so, one can repeat it with other basic models as well.

It is possible to train deep decision trees forests [1]. Then I believe that it is possible to train deep convolutional decision trees forests.

[1] https://arxiv.org/pdf/1702.08835

Re: Decision trees – the unreasonable power of nested decision rules

#88

Post by cubefox »

fooker wrote: post #74
cubefox wrote: post #65

Earlier quoted context omitted.

> Training a one bit neural network from scratch is apparently an unsolved problem though. It was until recently, but there is a new method which trains them directly without any floating point math, using "Boolean variation" instead of Newton/Leibniz differentiation: https://proceedings.neurips.cc/paper_files/paper/2024/hash/7...

Nice!

Unfortunately the paper seems to have been mostly overlooked. It has only a few citations. I think one practical issue is that that existing training hardware is optimized for floating point operations.

Re: Decision trees – the unreasonable power of nested decision rules

#89

Post by bawis »

levocardia wrote: post #78
jebarker wrote: post #49

The killer feature of DTs is how fast they can be. I worked very hard on a project to try and replace DT based classifiers with small NNs in a low latency application. NNs could achieve non-trivial gains in classification accuracy but remained two orders of magnitude higher latency at inference time.

Also, decision trees (but not their boosted or bagged variants) are easy (well, easy-ish) to port manually to an edge device that needs to run inference. Small vanilla NNs are as well, but many other popular "classical" ML algorithms are not.

>> but many other popular "classical" ML algorithms are not

Examples ?

Post reply on HN