Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we stopped using the mathematics that works

gfrm.in

Post reply on HN
Print view View original

21–30 of 46 posts

Re: Why we stopped using the mathematics that works

#21

Post by chbint »

jeffrallen wrote: post #3

Tldr: the author is annoyed at the Bitter Lesson. Join the crowd dude. It's still true, no matter how inconvenient it is.

I suspect his diagnostic is pretty accurate, though. The bitter lesson came up when deep learning was already mainstream. The text discusses how that happened, and it can be the case that convenience beats accuracy. Accuracy is an epistemic value, but current AI is largely driven by market values. If accuracy manages to get along, great, but other than that, market-laden convenience reigns. Commercially, it is often more convenient to even change the world in order to make it easier for our models (consider how we're willing to create special places without pedestrians or human-driven vehicles for autonomous vehicles as a "solution" for their shortcomings).

Re: Why we stopped using the mathematics that works

#22

Post by LoganDark »

jerf wrote: post #18
LoganDark wrote: post #7

Earlier quoted context omitted.

I think what they're saying is the methods used today are faster but have a lower ceiling, and that that's why they quickly took over but can only go so far.

That would be a hypothesis, not a fact. I'm not closed to it. You can check my comment history for frequent references to next-generation AIs that aren't architected like LLMs. But they're going to have to produce an AI of some sort that is better than the current ones, not hypothesize that it may be possible. We've got about 50 years of hypothesis about how wonderful such techniques may be and, by the new standards…

> That would be a hypothesis, not a fact.

I agree.

Re: Why we stopped using the mathematics that works

#23

Post by zozbot234 »

PaulHoule wrote: post #15
throwaway132448 wrote: post #4

I found the article confusing. Its premise seems to be that alternative methods to deep learning “work”, and only faded out due to other factors, yet keeps referencing scenarios in which they demonstrably failed to “work”. Such as: > In 2012, Alex Krizhevsky submitted a deep convolutional neural network to the ImageNet Large Scale Visual Recognition Challenge. It won by 9.8 percentage points over the nearest competit…

I think the worst thing about the golden age of symbolic AI was that there was never a systematic approach to reasoning about uncertainty. The MYCIN system was rather good at medical diagnostics and like other systems of the time had an ad-hoc procedure to deal with uncertainty which is essential in medical diagnosis. The problem is that is not enough to say "predicate A has a 80% of being true" but rather if you hav…

Symbolic AI ala Mycin and other expert systems didn't do anything that a modern database query engine can't do with far greater performance. The bottleneck is coming up with the set of rules that the system is to follow.

Re: Why we stopped using the mathematics that works

#24

Post by bpt3 »

notlenin wrote: post #19
NateEag wrote: post #10

Earlier quoted context omitted.

It means trying to figure out how to build an intelligence always loses to mindlessly brute-forcing problems with more compute: https://en.wikipedia.org/wiki/Bitter_lesson

unless you don't have unlimited compute, at which point you need other ideas https://arielche.net/bitter-lesson

Then train your model elsewhere and size it as appropriate for the runtime environment.

If that really isn't an option, then yes ML/AI isn't for you in this case.

Re: Why we stopped using the mathematics that works

#25

Post by andai »

notlenin wrote: post #19
NateEag wrote: post #10

Earlier quoted context omitted.

It means trying to figure out how to build an intelligence always loses to mindlessly brute-forcing problems with more compute: https://en.wikipedia.org/wiki/Bitter_lesson

unless you don't have unlimited compute, at which point you need other ideas https://arielche.net/bitter-lesson

I found this article a little weak, but there is an interesting parallel.

The 10,000 hours thing is encouraging because the amount of effort you put in as far more important than your natural ability.

... Until you get to the point where everyone is already working as hard as humanly possible, at which point natural ability becomes the sorting function again.

Re: Why we stopped using the mathematics that works

#26

Post by canjobear »

deckar01 wrote: post #8
throwaway132448 wrote: post #4

I found the article confusing. Its premise seems to be that alternative methods to deep learning “work”, and only faded out due to other factors, yet keeps referencing scenarios in which they demonstrably failed to “work”. Such as: > In 2012, Alex Krizhevsky submitted a deep convolutional neural network to the ImageNet Large Scale Visual Recognition Challenge. It won by 9.8 percentage points over the nearest competit…

It seems to be an indirect attempt to promote their GitHub project. They had Claude make them an “agent” using Bayesian modeling and Thompson sampling and now they are convinced they have heralded a new era of AI.

It reads to me like Claude wrote the article too.

Re: Why we stopped using the mathematics that works

#27

Post by danaris »

NateEag wrote: post #10
andai wrote: post #9

Earlier quoted context omitted.

This means money beats math?

It means trying to figure out how to build an intelligence always loses to mindlessly brute-forcing problems with more compute: https://en.wikipedia.org/wiki/Bitter_lesson

Well, it means that thus far trying to build an intelligence has lost out to brute forcing it with more compute.

There is nothing particular that suggests this is infinitely scalable.

Re: Why we stopped using the mathematics that works

#28

Post by psychoslave »

>This is the VHS-versus-Betamax dynamic, or TCP/IP versus the OSI model, or QWERTY versus every ergonomic alternative proposed since 1936.

QWERTY has many variants, and every single geopolitical institution have their own odious anti-ergonomic layout, it seems. So this case is somehow different to my mind. As a French native, I use Bépo.

Re: Why we stopped using the mathematics that works

#29

Post by canjobear »

andai wrote: post #11
furyofantares wrote: post #6

LLM-garbage article, ironically.

What makes you say that? Which LLM does it sound like to you?

> Not because the methods it displaced had stopped working, but because the money, the talent, and the prestige had moved elsewhere. The researchers who understood decision theory, Bayesian inference, and operations research didn’t lose their arguments. They lost their audience.

Re: Why we stopped using the mathematics that works

#30

Post by pron »

> This is the VHS-versus-Betamax dynamic, or TCP/IP versus the OSI model, or QWERTY versus every ergonomic alternative proposed since 1936. The technically superior solution loses to the solution that’s easier to deploy, easier to hire for, and good enough for the use cases that pay the bills.

Without commenting on the merit of the claims, the problem with this statement is that in many cases there is no universal "technical superiority", only tradeoffs. E.g. Betamax was technically superior in picture quality while VHS was technically superior in recording time, and more people preferred the latter technical superiority. When people say that the techinically superior approach lost in favour of convenience, what really happened is that their own personal technical preferences were in the minority. More people preferred an alternative that wasn't just "good enough" but technically better, only on a different axis.

Even if we suppose the author is right that his preferred approach yields better outputs, he acknowledges that constructing good inputs is harder. That's not technical superiority; it's a different tradeoff.

Post reply on HN