11–20 of 144 posts
The #1 thing that makes it ‘hard’ in real life is that nobody wants to make training and test sets. So we have 50,000 papers on the NIST digits but no insight into ‘would this work for a different problem?’ (Ironically the latter might have been exactly what academics would have needed to understand why these algorithms work!)
The #1 thing that makes it ‘hard’ in real life is that nobody wants to make training and test sets. So we have 50,000 papers on the NIST digits but no insight into ‘would this work for a different problem?’ (Ironically the latter might have been exactly what academics would have needed to understand why these algorithms work!)
Discussed at the time: Why is machine learning ‘hard’? - https://news.ycombinator.com/item?id=12936891 - Nov 2016 (88 comments)
Love that thread. The top comment is excellent: > Like picking hyperparamters - time and time again I've asked experts/trainers/colleagues: "How do I know what type of model to use? How many layers? How many nodes per layer? Dropout or not?" etc etc And the answer is always along the lines of "just try a load of stuff and pick the one that works best". > To me, that feels weird and worrying. It's like we don't yet un…
I’m deeply uncomfortable with such a method…but my models perform quite well. Note I spend a TON of time generating the right training data, so it’s not random.
The #1 thing that makes it ‘hard’ in real life is that nobody wants to make training and test sets. So we have 50,000 papers on the NIST digits but no insight into ‘would this work for a different problem?’ (Ironically the latter might have been exactly what academics would have needed to understand why these algorithms work!)
Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity.
I'm a bit sceptical of the exponentially harder debugging claim. First it looks polynomially harder for the given example :p. Second other engineering domains arguably have additional dimensions which correspond to the machine learning ones mentioned in the article. The choice of which high level algorithm to implement is another dimension to traditional software engineering that seemingly exists and corresponds to t…
Earlier quoted context omitted.
Love that thread. The top comment is excellent: > Like picking hyperparamters - time and time again I've asked experts/trainers/colleagues: "How do I know what type of model to use? How many layers? How many nodes per layer? Dropout or not?" etc etc And the answer is always along the lines of "just try a load of stuff and pick the one that works best". > To me, that feels weird and worrying. It's like we don't yet un…
The hammer analogy doesn't make much sense because for a hammer we can actually use our scientific knowledge to compute the best possible way to hold the tool, and we can make instruments that are better than hammers, like pneumatic hammers, pile drivers, etc. With your argument, we would be stuck with the good old, but basic hammer for the rest of time.
And in any case, I'm not saying we shouldn't search for deep understanding of what hyperparameters work on a first try, I'm just saying there's a good chance that even if the principles are fully discovered, it may be that calculating using those principles is more expensive than a bunch of experimentation and won't matter in the end.
That's the trick about science, it's more about finding the right question to answer than how to find answers, and often times the best questions only become apparent afterwards.
There are some deep connections between data compression and generalized learning, both at the statistical level and even lower at the algorithmic level (see Solomonoff induction).
For a specific example at the statistical level, suppose you fit a linear trendline to some data points using OLS. Now compute the standard deviation of the residual terms for each data point, and using the CDF of the normal distribution, for each residual, map its value into the interval [0, 1]. Sum together the logarithms of the trendline coefficients, the standard deviation, and the normalized residuals. This value is approximately proportional to "sizeof(data | model) + sizeof(model)". It represents how well you compressed the data using the OLS model.
But now suppose you plot the distribution of the residuals and find out that they do not in fact resemble a Gaussian distribution. This is a problem because our model assumed the error terms were distributed normally, and since this is not the case, our compression is suboptimal.
So you back out some function f that closely maps between the uniform distribution on [0, 1] and the distribution that the residuals form and use this f to define a new model: yᵢ = m*xᵢ + b + εᵢ, with εᵢ distributed according to f(x;Θ), Θ being a parameter vector. When you sum the logarithms again, you will find that the new total is smaller than the original total obtained using OLS. The new trend line coefficients will slightly mess up the residual distribution again, so iterate on this process until you've converged on stable values for m, b, and Θ.
At the algorithmic level, the recommendation to use compression as a model selection guide applies even to LLMs, but it's a bit harder to use in practice because "sizeof(data | model) + sizeof(model)" isn't the entire story here. Suppose you had a "perfect" language model. In this case, you would achieve minimization of K(data | training data), where K is Kolmogorov complexity. In practice, what is being minimized with each new LLM version is "sizeof(data | LLM model) + sizeof(training data | LLM model) + sizeof(LLM model)". You can assume "sizeof(LLM model)" is the smallest Turing machine equivalent to the LLM program.
Discussed at the time: Why is machine learning ‘hard’? - https://news.ycombinator.com/item?id=12936891 - Nov 2016 (88 comments)
Love that thread. The top comment is excellent: > Like picking hyperparamters - time and time again I've asked experts/trainers/colleagues: "How do I know what type of model to use? How many layers? How many nodes per layer? Dropout or not?" etc etc And the answer is always along the lines of "just try a load of stuff and pick the one that works best". > To me, that feels weird and worrying. It's like we don't yet un…
Not really though. In engineering, you have heuristics, even if you don't know why they work. In the case of deep learning / AI, there seems to be very little in the way of built up heuristic knowledge - it's just "try stuff and see what works for every problem".
Earlier quoted context omitted.
Love that thread. The top comment is excellent: > Like picking hyperparamters - time and time again I've asked experts/trainers/colleagues: "How do I know what type of model to use? How many layers? How many nodes per layer? Dropout or not?" etc etc And the answer is always along the lines of "just try a load of stuff and pick the one that works best". > To me, that feels weird and worrying. It's like we don't yet un…
I do a lot of model tuning and I’m almost ashamed to say I tell GPT what performance I’m aiming for and have it generate the hyper parameters (as in just literally give me a code block). Then I see what works, tell GPT, and try again. I’m deeply uncomfortable with such a method…but my models perform quite well. Note I spend a TON of time generating the right training data, so it’s not random.
"I do a lot of X and I'm almost ashamed to say I tell GPT Y then I see if it works and try again".