Decision trees – the unreasonable power of nested decision rules • Hacker News Board

Re: Decision trees – the unreasonable power of nested decision rules

#21

Post by hansvm » Sun, Mar 01, 2026, 1:13 PM UTC

wodenokoto wrote: post #16

lokimedes wrote: post #14

When I worked at CERN around 2010, Boosted Decision Trees were the most popular classifier, exactly due to the (potential for) explainability along with its power of expression. We had a cultural aversion for neural networks back then, especially if the model was used in physics analysis directly. Times have changed…

Are boosted decision trees the same as a boosted random forest?

The training process differs, but the resulting model only differs in data rather than code -- you evaluate a bunch of trees and add them up.

For better or for worse (usually for better), boosted decision trees work harder to optimize the tree structure for a given problem. Random forests rely on enough trees being good enough.

Ignoring tree split selection, one technique people sometimes do makes the two techniques more related -- in gradient boosting, once the splits are chosen it's a sparse linear algebra problem to optimize the weights/leaves (iterative if your error is not MSE). That step would unify some part of the training between the two model types.

Re: Decision trees – the unreasonable power of nested decision rules

#22

Post by srean » Sun, Mar 01, 2026, 1:19 PM UTC

zelphirkalt wrote: post #20

srean wrote: post #15

Earlier quoted context omitted.

What benefit does numpy or dataframes bring to decision tree logic over what is available in Guile already ? Honest question. Guile like languages are very well suited for decision trees, because manipulating and operating on trees is it's mother tongue. Only thing that would be a bit more work would be to compile the decision tree into machine code. Then one doesn't have traverse a runtime structure, the former bein…

I think data frames are quite memory efficient and can store non-uniform data types (as can vectors in Guile). Generally, a ton of work has gone into making operations on data frames fast. I don't think a normal vector or multi-dimensional array can easily compete. Data frames are probably also compiled to some quite efficient machine code. Not sure whether Guile's native data structures can match that. Maybe they ca…

For linear algebraic transformation applied to several rows at once, I wholeheartedly agree.

Not so convinced about decision trees though (that process one row at a time).

Yeah, unless you had to deal with arbitrarily large integer features, Guile integers would come with a big efficiency hit.

Re: Decision trees – the unreasonable power of nested decision rules

#23

Post by srean » Sun, Mar 01, 2026, 1:48 PM UTC

lokimedes wrote: post #14

When I worked at CERN around 2010, Boosted Decision Trees were the most popular classifier, exactly due to the (potential for) explainability along with its power of expression. We had a cultural aversion for neural networks back then, especially if the model was used in physics analysis directly. Times have changed…

> Times have changed…

This makes me a little concerned -- the use of parameters rich opaque models in Physics.

Ptolemaic system achieved a far better fit of planetary motion (over the Copernican system) because his was a universal approximator. Epicyclic system is a form of Fourier analysis and hence can fit any smooth periodic motion. But the epicycles were not the right thing to use to work out the causal mechanics, in spite of being a better fit empirically.

In Physics we would want to do more than accurate curve fitting.

Re: Decision trees – the unreasonable power of nested decision rules

#24

Post by ekjhgkejhgk » Sun, Mar 01, 2026, 1:58 PM UTC

srean wrote: post #19

A 'secret weapon' that has served me very well for learning classifiers is to first learn a good linear classifier. I am almost hesitant to give this away (kidding). Use the non-thresholded version of that linear classifier output as one additional feature-dimension over which you learn a decision tree. Then wrap this whole thing up as a system of boosted trees (that is, with more short trees added if needed). One of…

> (ii) linear functions are terrible where equi-label regions have a partitioned structure.

Could you explain what "equi-label regions having a partitioned structure" mean?

Re: Decision trees – the unreasonable power of nested decision rules

#25

Post by smokel » Sun, Mar 01, 2026, 1:59 PM UTC

fooker wrote: post #6

Fun fact - single bit neural networks are decision trees. In theory, this means you can 'compile' most neural networks into chains of if-else statements but it's not well understood when this sort of approach works well.

> single bit neural networks are decision trees.

I didn't exactly understood what was meant here, so I went out and read a little. There is an interesting paper called "Neural Networks are Decision Trees" [1]. Thing is, this does not imply a nice mapping of neural networks onto decision trees. The trees that correspond to the neural networks are huge. And I get the idea that the paper is stretching the concept of decision trees a bit.

Also, I still don't know exactly what you mean, so would you care to elaborate a bit? :)

[1] https://arxiv.org/pdf/2210.05189

Re: Decision trees – the unreasonable power of nested decision rules

#26

Post by srean » Sun, Mar 01, 2026, 2:06 PM UTC

ekjhgkejhgk wrote: post #24

srean wrote: post #19

A 'secret weapon' that has served me very well for learning classifiers is to first learn a good linear classifier. I am almost hesitant to give this away (kidding). Use the non-thresholded version of that linear classifier output as one additional feature-dimension over which you learn a decision tree. Then wrap this whole thing up as a system of boosted trees (that is, with more short trees added if needed). One of…

> (ii) linear functions are terrible where equi-label regions have a partitioned structure. Could you explain what "equi-label regions having a partitioned structure" mean?

I missed a word "recursively", that I have edited in my original comment now.

Consider connected regions in the domain that have the same label. Much like countries on a political map. The situation where this has a short description in terms of recursive subdivision of space, is what I am calling a partitioned structure. It's really rather tautological.

Re: Decision trees – the unreasonable power of nested decision rules

#27

Post by lokimedes » Sun, Mar 01, 2026, 2:10 PM UTC

srean wrote: post #23

lokimedes wrote: post #14

When I worked at CERN around 2010, Boosted Decision Trees were the most popular classifier, exactly due to the (potential for) explainability along with its power of expression. We had a cultural aversion for neural networks back then, especially if the model was used in physics analysis directly. Times have changed…

> Times have changed… This makes me a little concerned -- the use of parameters rich opaque models in Physics. Ptolemaic system achieved a far better fit of planetary motion (over the Copernican system) because his was a universal approximator. Epicyclic system is a form of Fourier analysis and hence can fit any smooth periodic motion. But the epicycles were not the right thing to use to work out the causal mechanics…

If you sum up experimental physics into one heuristic it is “avoid fooling yourself with assumptions” - I left physics over a decade ago, but I feel confident that physicists still work hard to understand what they observe and don’t let LLMs have all the fun. If there’s one field of science where the scientists are legitimately allowed to go all the way back to basics, it’s elementary particle physics.

Re: Decision trees – the unreasonable power of nested decision rules

#28

Post by huqedato » Sun, Mar 01, 2026, 2:13 PM UTC

Random forests on the same site: https://mlu-explain.github.io/random-forest/

Re: Decision trees – the unreasonable power of nested decision rules

#29

Post by srean » Sun, Mar 01, 2026, 2:24 PM UTC

lokimedes wrote: post #27

srean wrote: post #23

Earlier quoted context omitted.

> Times have changed… This makes me a little concerned -- the use of parameters rich opaque models in Physics. Ptolemaic system achieved a far better fit of planetary motion (over the Copernican system) because his was a universal approximator. Epicyclic system is a form of Fourier analysis and hence can fit any smooth periodic motion. But the epicycles were not the right thing to use to work out the causal mechanics…

If you sum up experimental physics into one heuristic it is “avoid fooling yourself with assumptions” - I left physics over a decade ago, but I feel confident that physicists still work hard to understand what they observe and don’t let LLMs have all the fun. If there’s one field of science where the scientists are legitimately allowed to go all the way back to basics, it’s elementary particle physics.

In general I would agree. I think it holds true at the highest levels.

What worries me is the noticeable uptick of presentations of the sort -- look ma better fit ... deep neural nets. These are mostly by more junior folks, but not necessarily. I have been in the audience in many.

These and the uptick in research proposals funded by providers of infra for such DNNs. I have been in the audience of many.

A charitable read could be that they just want the money and would do the principled thing.

Re: Decision trees – the unreasonable power of nested decision rules

#30

Post by lioeters » Sun, Mar 01, 2026, 2:29 PM UTC

smokel wrote: post #25

fooker wrote: post #6

Fun fact - single bit neural networks are decision trees. In theory, this means you can 'compile' most neural networks into chains of if-else statements but it's not well understood when this sort of approach works well.

> single bit neural networks are decision trees. I didn't exactly understood what was meant here, so I went out and read a little. There is an interesting paper called "Neural Networks are Decision Trees" [1]. Thing is, this does not imply a nice mapping of neural networks onto decision trees. The trees that correspond to the neural networks are huge . And I get the idea that the paper is stretching the concept of de…

Closest thing I found was:

Single Bit Neural Nets Did Not Work - https://fpga.mit.edu/videos/2023/team04/report.pdf

> We originally planned to make and train a neural network with single bit activations, weights, and gradients, but unfortunately the neural network did not train very well. We were left with a peculiar looking CPU that we tried adapting to mine bitcoin and run Brainfuck.