Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Decision trees – the unreasonable power of nested decision rules

mlu-explain.github.io

Post reply on HN
Print view View original

51–60 of 89 posts

Re: Decision trees – the unreasonable power of nested decision rules

#51

Post by [deleted] »

srean wrote: post #23
lokimedes wrote: post #14

When I worked at CERN around 2010, Boosted Decision Trees were the most popular classifier, exactly due to the (potential for) explainability along with its power of expression. We had a cultural aversion for neural networks back then, especially if the model was used in physics analysis directly. Times have changed…

> Times have changed… This makes me a little concerned -- the use of parameters rich opaque models in Physics. Ptolemaic system achieved a far better fit of planetary motion (over the Copernican system) because his was a universal approximator. Epicyclic system is a form of Fourier analysis and hence can fit any smooth periodic motion. But the epicycles were not the right thing to use to work out the causal mechanics…

[deleted]

Re: Decision trees – the unreasonable power of nested decision rules

#52

Post by lokimedes »

ekjhgkejhgk wrote: post #48
lokimedes wrote: post #14

When I worked at CERN around 2010, Boosted Decision Trees were the most popular classifier, exactly due to the (potential for) explainability along with its power of expression. We had a cultural aversion for neural networks back then, especially if the model was used in physics analysis directly. Times have changed…

I used to be in physics but theory, not experiment. I have experience at work with decision trees in a different field. I've always thought that the idea that decision trees are "explainable" is very overstated. The moment that you go past a couple of levels in depth, it becomes an un-interpretable jungle. I've actually done the exercise of inspecting how a 15-depth decision trees makes decision, and I found it impos…

I completely agree, as you may infer from my comment. The second multivariate models are relevant we effectively trade explainability for discrimination power. If your decision tree/model needs to be large enough to warrant SGD or similar optimization techniques, it is pretty much a fantasy to ever analyze it formally.

My second job after physics was AI for defense, and boy is the dream of explainable AI alive there.

Honesty anyone who “needs” AI to be understandable by dissection, suffers from control issues :)

Re: Decision trees – the unreasonable power of nested decision rules

#53

Post by fooker »

smokel wrote: post #25
fooker wrote: post #6

Fun fact - single bit neural networks are decision trees. In theory, this means you can 'compile' most neural networks into chains of if-else statements but it's not well understood when this sort of approach works well.

> single bit neural networks are decision trees. I didn't exactly understood what was meant here, so I went out and read a little. There is an interesting paper called "Neural Networks are Decision Trees" [1]. Thing is, this does not imply a nice mapping of neural networks onto decision trees. The trees that correspond to the neural networks are huge . And I get the idea that the paper is stretching the concept of de…

> I still don't know exactly what you mean

Straight forward quantization, just to one bit instead of 8 or 16 or 32. Training a one bit neural network from scratch is apparently an unsolved problem though.

> The trees that correspond to the neural networks are huge.

Yes, if the task is inherently 'fuzzy'. Many neural networks are effectively large decision trees in disguise and those are the ones which have potential with this kind of approach.

Re: Decision trees – the unreasonable power of nested decision rules

#54

Post by fooker »

Almondsetat wrote: post #7
fooker wrote: post #6

Fun fact - single bit neural networks are decision trees. In theory, this means you can 'compile' most neural networks into chains of if-else statements but it's not well understood when this sort of approach works well.

Do you know of any software that does this? Or any papers on the matter? It could be a fun weekend project

I think any quantization approach should work, not an expert on this.

Re: Decision trees – the unreasonable power of nested decision rules

#55

Post by srean »

EGreg wrote: post #33

Isn’t that exactly how humans (and even animals) operate? Human societies look for actual major correlations and establish classifications. Except with scientific-minded humans, we often also want, to know the why behind the correlations. David Hume got involved w that… https://brainly.com/question/50372476 Let me ask a provocative question. What, ultimately, is the difference between knowledge and bias?

To a certain degree yes.

https://en.wikipedia.org/wiki/Esagil-kin-apli

In this Mesopotamian text, diagnostic rules are structured as a nest of if then else rules. So I have been told, not that I have read it myself.

Re: Decision trees – the unreasonable power of nested decision rules

#56

Post by whatever1 »

srean wrote: post #19

A 'secret weapon' that has served me very well for learning classifiers is to first learn a good linear classifier. I am almost hesitant to give this away (kidding). Use the non-thresholded version of that linear classifier output as one additional feature-dimension over which you learn a decision tree. Then wrap this whole thing up as a system of boosted trees (that is, with more short trees added if needed). One of…

If you think about it, this what reinforcement learning folks are doing. They use the vanilla state and then they lift it to the observed state by doing some additional calculation with the original state data.

For example you start with the raw coordinates of snake in a snake game, but you now can calculate how many escape routes the snake has, and train on it.

Re: Decision trees – the unreasonable power of nested decision rules

#57

Post by mistrial9 »

jjcc wrote: post #39
getpokedagain wrote: post #35

I worked (professionally) on a product a few years ago based upon decision tree and random forest classifiers. I had no background in the math and had to learn this stuff which has payed dividends as llms and AI have become hyped. This is one of the best explanations I've seen and has me super nostalgic for that project. Gonna try to cook up something personal. It's amazing how people are now using regression models…

I worked on a product which was the best ID reader in the world at the time 25 years ago. The OCR engine was based on Decision tree and "Random Forest" (I suspect the name did exist) with only 3 trees. It was very effective as a secret weapon of the competitiveness. I tried to train a NN with a framework called SNNS(Stuttgart Neural Network Simulator) as the 4th tree complement to the existing 3. Today, hand writing…

in the interest of understanding, is there any code or similar for the approach? does that OCR run anywhere today?

Re: Decision trees – the unreasonable power of nested decision rules

#58

Post by selimthegrim »

srean wrote: post #19

A 'secret weapon' that has served me very well for learning classifiers is to first learn a good linear classifier. I am almost hesitant to give this away (kidding). Use the non-thresholded version of that linear classifier output as one additional feature-dimension over which you learn a decision tree. Then wrap this whole thing up as a system of boosted trees (that is, with more short trees added if needed). One of…

Is this not the heart of the IRM paper by Arjovsky, Bottou et al.?

Re: Decision trees – the unreasonable power of nested decision rules

#60

Post by AndrewKemendo »

whatever1 wrote: post #56
srean wrote: post #19

A 'secret weapon' that has served me very well for learning classifiers is to first learn a good linear classifier. I am almost hesitant to give this away (kidding). Use the non-thresholded version of that linear classifier output as one additional feature-dimension over which you learn a decision tree. Then wrap this whole thing up as a system of boosted trees (that is, with more short trees added if needed). One of…

If you think about it, this what reinforcement learning folks are doing. They use the vanilla state and then they lift it to the observed state by doing some additional calculation with the original state data. For example you start with the raw coordinates of snake in a snake game, but you now can calculate how many escape routes the snake has, and train on it.

That’s correct

I’ve spent most of the last 20 years doing reinforcement learning and it is exceptionally simple conceptually

The challenge is data acquisition and the right types of process frameworks

Post reply on HN