Wow. This page is actually a product of LLM [0]. So they can produce useful stuff after all :)
Decision trees – the unreasonable power of nested decision rules
61–70 of 89 posts
Re: Decision trees – the unreasonable power of nested decision rules
#62Wow. This page is actually a product of LLM [0]. So they can produce useful stuff after all :) [0]: https://news.ycombinator.com/item?id=47195123
No, you misread
Re: Decision trees – the unreasonable power of nested decision rules
#63A 'secret weapon' that has served me very well for learning classifiers is to first learn a good linear classifier. I am almost hesitant to give this away (kidding). Use the non-thresholded version of that linear classifier output as one additional feature-dimension over which you learn a decision tree. Then wrap this whole thing up as a system of boosted trees (that is, with more short trees added if needed). One of…
Is this not the heart of the IRM paper by Arjovsky, Bottou et al.?
Oh theirs is a far more sophisticated and deeper idea, to look for invariant representations.
Mine is a quick but effective practical hack fuelled by a little bit of insight.
Re: Decision trees – the unreasonable power of nested decision rules
#64Earlier quoted context omitted.
> single bit neural networks are decision trees. I didn't exactly understood what was meant here, so I went out and read a little. There is an interesting paper called "Neural Networks are Decision Trees" [1]. Thing is, this does not imply a nice mapping of neural networks onto decision trees. The trees that correspond to the neural networks are huge . And I get the idea that the paper is stretching the concept of de…
> I still don't know exactly what you mean Straight forward quantization, just to one bit instead of 8 or 16 or 32. Training a one bit neural network from scratch is apparently an unsolved problem though. > The trees that correspond to the neural networks are huge. Yes, if the task is inherently 'fuzzy'. Many neural networks are effectively large decision trees in disguise and those are the ones which have potential…
> Training a one bit neural network from scratch is apparently an unsolved problem though.
I don't think it's correct to call it unsolved. The established methods are much less efficient than those for "regular" neural nets but they do exist.
Also note that the usual approach when going binary is to make the units stochastic. https://en.wikipedia.org/wiki/Boltzmann_machine#Deep_Boltzma...
Re: Decision trees – the unreasonable power of nested decision rules
#65Earlier quoted context omitted.
> single bit neural networks are decision trees. I didn't exactly understood what was meant here, so I went out and read a little. There is an interesting paper called "Neural Networks are Decision Trees" [1]. Thing is, this does not imply a nice mapping of neural networks onto decision trees. The trees that correspond to the neural networks are huge . And I get the idea that the paper is stretching the concept of de…
> I still don't know exactly what you mean Straight forward quantization, just to one bit instead of 8 or 16 or 32. Training a one bit neural network from scratch is apparently an unsolved problem though. > The trees that correspond to the neural networks are huge. Yes, if the task is inherently 'fuzzy'. Many neural networks are effectively large decision trees in disguise and those are the ones which have potential…
> Training a one bit neural network from scratch is apparently an unsolved problem though.
It was until recently, but there is a new method which trains them directly without any floating point math, using "Boolean variation" instead of Newton/Leibniz differentiation:
https://proceedings.neurips.cc/paper_files/paper/2024/hash/7...
Re: Decision trees – the unreasonable power of nested decision rules
#66A 'secret weapon' that has served me very well for learning classifiers is to first learn a good linear classifier. I am almost hesitant to give this away (kidding). Use the non-thresholded version of that linear classifier output as one additional feature-dimension over which you learn a decision tree. Then wrap this whole thing up as a system of boosted trees (that is, with more short trees added if needed). One of…
I think it's worth mentioning, that the achilles heel of DT, is in fact, data (more specifically feature) engineering. If one does not spend significant time cleaning and engineering the features, the results would be much worse than, say a "black box" model, like NN. This is the catch. Ironically, NN can detect such latent features, but very difficult to interpret why.
This varies so wildly from domain to domain. Highly structured data (time series, photos, audio, etc.) typically has a metric boatload of feature extraction methodology. Neural networks often draw on and exploit that structure (i.e. convolutions). You could even get some pretty good results on manually-extracted neural-network-esque features handed off to a random forest. This heuristic begins to fall off with deep learning though, which imo, is a precursor to LLMs and showed that emergent complexity is possible with machine learning.
But non-structured data? Pretty pointless to hand off to a neural network imo.
Re: Decision trees – the unreasonable power of nested decision rules
#67Wow. This page is actually a product of LLM [0]. So they can produce useful stuff after all :) [0]: https://news.ycombinator.com/item?id=47195123
No, you misread
And you are absolutely correct. I've seen the DT page thanks to the linked HN submission (actually comment [1]. And incorrectly associated the DT article incorrectly today. Thank you.
Re: Decision trees – the unreasonable power of nested decision rules
#68Earlier quoted context omitted.
I think it's worth mentioning, that the achilles heel of DT, is in fact, data (more specifically feature) engineering. If one does not spend significant time cleaning and engineering the features, the results would be much worse than, say a "black box" model, like NN. This is the catch. Ironically, NN can detect such latent features, but very difficult to interpret why.
This varies so wildly from domain to domain. Highly structured data (time series, photos, audio, etc.) typically has a metric boatload of feature extraction methodology. Neural networks often draw on and exploit that structure (i.e. convolutions). You could even get some pretty good results on manually-extracted neural-network-esque features handed off to a random forest. This heuristic begins to fall off with deep l…
Ah! Your comment helped me understand the parent comment so much more. I thought it was more about data hygiene needs.
Yes a DT on raw pixel values, or a DT on raw time values will in general be quite terrible.
That said the convolutional structure is hard coded in those neural nets, only the weights are learned. It is not that the network discovered on its own that convolutions are a good idea. So NNs too really (damn autocorrect, it's rely, rely) on human insight and structuring upon which they can then build over.
Re: Decision trees – the unreasonable power of nested decision rules
#69Decision trees are underrated in the age of deep learning. They're interpretable, fast, and often good enough.
I've been using a scoring system for website analysis that's essentially a decision tree under the hood. Does the site have a meta description? Does it load in under 3 seconds? Is it mobile responsive? Each check produces a score, the tree aggregates them. Users understand why they got their score because the logic is transparent.
Try explaining why a neural network rated their website 73/100. Decision trees make that trivial.
Re: Decision trees – the unreasonable power of nested decision rules
#70Decision trees are underrated in the age of deep learning. They're interpretable, fast, and often good enough. I've been using a scoring system for website analysis that's essentially a decision tree under the hood. Does the site have a meta description? Does it load in under 3 seconds? Is it mobile responsive? Each check produces a score, the tree aggregates them. Users understand why they got their score because th…
You sir are keeping alive an old tradition. 1K BC old.
https://en.wikipedia.org/wiki/Esagil-kin-apli#The_Sakikk%C5%...