The Sell ∀ ∃ as ∃ ∀ Scam
win-vector.com
The Sell ∀ ∃ as ∃ ∀ Scam
1–10 of 148 posts
Re: The Sell ∀ ∃ as ∃ ∀ Scam
#2Since there are certainly people unfamiliar with the symbols, here is the Wikipedia article on them:
Re: The Sell ∀ ∃ as ∃ ∀ Scam
#3This is a clever argument
Re: The Sell ∀ ∃ as ∃ ∀ Scam
#4This argument is disingenuous. Hyperparameter optimization is not in the same category as prompt engineering, like at all.
Also no one claims like half of the things the article claims people clain.
Re: The Sell ∀ ∃ as ∃ ∀ Scam
#5>To conclude: one must have different standards for developing systems than for testing, deploying, or using systems. Or: testing on your training data is a common way to cheat, but so is training on your test data.
Isn't this already a solved problem? Every reasonable paper on ML separates their test data from their validation data already.
Re: The Sell ∀ ∃ as ∃ ∀ Scam
#6It's a good point, I'd consider it (overfitting) a pitfall or common mistake in ML rather than the only mode. I'd agree that most ML models and almost all state-of-the-art are over-fit to the point of being useless, but that's not an inevitability.
Re: The Sell ∀ ∃ as ∃ ∀ Scam
#7Since there are certainly people unfamiliar with the symbols, here is the Wikipedia article on them: https://en.wikipedia.org/wiki/Quantifier_(logic)
The article provides abbreviated definitions that might be less confusing to follow than the comprehensive Wikipedia page. But, I agree that calling them out a bit more clearly in the article would have been helpful.
Re: The Sell ∀ ∃ as ∃ ∀ Scam
#8>To conclude: one must have different standards for developing systems than for testing, deploying, or using systems. Or: testing on your training data is a common way to cheat, but so is training on your test data. Isn't this already a solved problem? Every reasonable paper on ML separates their test data from their validation data already.
That in no way prevents overfitting though hyperparameter optimization / graduate student descent. All the common benchmarks, by definition of being a common benchmark, are susceptible to overfitting
Re: The Sell ∀ ∃ as ∃ ∀ Scam
#9I can walk and chew bubble gum at the same time: on one hand, yes, there's certainly a lot of Kool-Aid being drank by the AI folks. Even on HN, I constantly argue with people that genuinely think LLMs are some kind of magical black box that contain "knowledge" or "intelligence" or "meaning" when in reality, it's just a very fancy Markov chain. And on the other hand, I think that language interfaces are probably the next big leap in how we interact with our computers, but more to the point of the article:
> To conclude: one must have different standards for developing systems than for testing, deploying, or using systems.
In my opinion, you unfortunately will never (and, in fact could never) have reliable development and testing standards when designing purely stochastic systems like, e.g., large language models. Intuitively, the fact that these are stochastic systems is why we need things like hyper-parameters, fiddling with seeds, and prompt engineering.
Re: The Sell ∀ ∃ as ∃ ∀ Scam
#10If I understand correctly, the meat of the argument is "that is a system for every (∀) task, there exists (∃) a setting that gives the correct answer for that one task."
My understanding of this (correct me if I'm wrong) is that the scam is convincing users that GPT-X can do anything with say, the correct prompts.
This argument misses the mark for me. It's not that it solves all the problems, it's that the problems it does solve is economically impactful. Significantly economically impactful in some cases- obvious examples of call centers and first-line customer support.