Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will scaling work?

dwarkeshpatel.com

Post reply on HN
Print view View original

121–130 of 289 posts

Re: Will scaling work?

#121

Post by johngossman »

HarHarVeryFunny wrote: post #96
bee_rider wrote: post #60

Earlier quoted context omitted.

The internet did change things pretty dramatically. Productivity at information communication tasks just isn’t the entire economy. I think we are massively more productive. Some of the biggest new companies are ad companies (Google, Facebook), or spend a ton of their time designing devices that can’t be modified by their users (Apple, Microsoft). Even old fashioned companies like tractor and train companies have time…

> The internet did change things pretty dramatically. For sure - I grew up in the mid-late 70s having to walk to the library to research stuff for homework, parents having to use the yellow-pages to find things, etc. Maybe smartphones are more of a game changer than desk-bound internet though - a global communication device in your pocket that'll give you driving directions, etc, etc. BUT ... does the world really FE…

I once asked my mom, who grew up in the 1930s (aside: feels increasingly necessary to specific 19--), what was the biggest technological change she had seen in her lifetime. Her immediate answer was 'indoor plumbing.' But her next answer was the cellphone. She said cars and trains weren't vastly different from when she was a kid, she almost never went on a plane, and that people spent a lot of time watching the TV and listening to the radio, but they used their cellphones more and for far more things.

Re: Will scaling work?

#122

Post by HarHarVeryFunny »

bee_rider wrote: post #108
HarHarVeryFunny wrote: post #96

Earlier quoted context omitted.

> The internet did change things pretty dramatically. For sure - I grew up in the mid-late 70s having to walk to the library to research stuff for homework, parents having to use the yellow-pages to find things, etc. Maybe smartphones are more of a game changer than desk-bound internet though - a global communication device in your pocket that'll give you driving directions, etc, etc. BUT ... does the world really FE…

I don’t know really, I was a kid in the 90’s. This is a bit far from the economic aspect, but the world currently seemed to be utterly suffused with a looming sense of dread, I think because we have, or know other people have, news notifications in their pockets telling us all about how bad things are. I don’t remember that feeling from the 90’s, but then, I was a kid. And of course before that there was the constant…

Yes - internet "news" is hardly a positive.

I grew up in the UK, so news was mainly from the BBC which was pretty decent although bad news (e.g. IRA bombings) was still front and center. US TV news doesn't even pretend/try to be unbiased and is all about shock value, reinforcing their viewers political beliefs and of course advertizing (which the BBC didn't have, being state funded).

Internet takes bad news and misinformation to a whole new and massively distorted level.

I gave up watching TV many years ago (nowadays primarily YouTube & Netflix for entertainment), and mostly just skim headlines (e.g. Google news) to get an idea of what's going on.

Re: Will scaling work?

#123

Post by sgt101 »

>Here’s one of the many astounding finds in Microsoft Research’s Sparks of AGI paper. They found that GPT-4 could write the LaTex code to draw a unicorn.

a lot of people have tried to replicate this, I have tried. It's very hard to get GPT-4 to draw a unicorn, also asking it to draw an upside down unicorn is even harder.

Re: Will scaling work?

#124

Post by johngossman »

toast0 wrote: post #105
HarHarVeryFunny wrote: post #96

Earlier quoted context omitted.

> The internet did change things pretty dramatically. For sure - I grew up in the mid-late 70s having to walk to the library to research stuff for homework, parents having to use the yellow-pages to find things, etc. Maybe smartphones are more of a game changer than desk-bound internet though - a global communication device in your pocket that'll give you driving directions, etc, etc. BUT ... does the world really FE…

I would say that it feels different because the internet / smartphones are more about giving everyone access to inexpensive, high bandwidth, communication (nearly) everywhere. But high bandwidth communications have been available everywhere for a long time, if you had a need and were willing to pay for it --- tv news would bounce signals off a satelite for on scene reports, etc.

It does feel different, but I don't think it's the bandwidth, or even the availability. A newspaper is high bandwidth and fairly inexpensive and ubiquitous but also fairly high latency. The evening TV news was only once a day until the 80s. One big change I noticed was 24-hour news. Suddenly, it felt important to know about things immediately. The web was different because it was interactive--both in the sense that you could swiftly switch between information sources and then in the social media sense that everybody could participate, even if participation meant flame wars.

And historically, TV news isn't that old, especially the 24-hour variety. The Apollo landings and Vietnam War are often cited as landmarks in TV news, where for the first time large numbers of people watched things as they occurred. But it's only about 25 years from those events to Netscape Navigator, where the web became widely available (at least in the developed world). That's a long time in most people's lives, but I wouldn't be surprised if future historians will see TV as something like an early, one-way Internet.

Re: Will scaling work?

#125

Post by lossolo »

A few more interesting papers not mentioned in the article:

"Faith and Fate: Limits of Transformers on Compositionality"

https://arxiv.org/abs/2305.18654

"Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks":

https://arxiv.org/abs/2311.09247

"Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve"

https://arxiv.org/abs/2309.13638

"Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models"

https://arxiv.org/abs/2311.00871

Re: Will scaling work?

#126

Post by hardwaregeek »

slibhb wrote: post #49

The best analogy for LLMs (up to and including AGI) is the internet + google search. Imagine explaining the internet/google to someone in 1950. That person might say "Oh my god, everything will change! Instantaneous, cheap communication! The world's information available at light speed! Science will accelerate, productivity will explode!" And yet, 70 years later, things have certainly changed, but we're living in the…

It's important to remember that the internet is still very very new. Like the generation of digital natives are barely in adulthood. Sure, it's existed in some form for about 40 years, but most of the world didn't have access for the longest time. I wouldn't be surprised if we see massive changes in the next 20 years from the people who grew up on the web (specifically people outside the United States and Europe, where access was harder for a long time)

Re: Will scaling work?

#127

Post by archon1410 »

skippyboxedhero wrote: post #116
espadrine wrote: post #46

Earlier quoted context omitted.

Demis Hassabis of Deepmind echoes a similar sentiment[0]: > I still think there are missing things with the current systems. […] I regard it a bit like the Industrial Revolution where there was all these amazing new ideas about energy and power and so on, but it was fueled by the fact that there were dead dinosaurs, and coal and oil just lying in the ground. Imagine how much harder the Industrial Revolution would hav…

His view of the Industrial Revolution is completely wrong. Societies pre-IR had multiple periods where energy usage increased significantly, some of them based specifically around coal. No IR. Early IR was largely based around the usage of water power, not coal. IR was pure innovation, people being able to imagine and create the impossible, it was going straight to nuclear already. Ironically, someone who is an innov…

Has the narrative moved on? The historian and blogger Bret Devereaux presents a view on a 2022 blog post that seems to back up what the Deepmind CEO is saying.

> The specificity matters here because each innovation in the chain required not merely the discovery of the principle, but also the design and an economically viable use-case to all line up in order to have impact.

https://acoup.blog/2022/08/26/collections-why-no-roman-indus...

Re: Will scaling work?

#128

Post by red75prime »

arketyp wrote: post #59
slibhb wrote: post #49

The best analogy for LLMs (up to and including AGI) is the internet + google search. Imagine explaining the internet/google to someone in 1950. That person might say "Oh my god, everything will change! Instantaneous, cheap communication! The world's information available at light speed! Science will accelerate, productivity will explode!" And yet, 70 years later, things have certainly changed, but we're living in the…

My take on this is that much of work and problem solving is about understanding the problem. So I think human abilities will remain the bottleneck. I pose this thought experiment: Is it possible to design an AI system for a monkey which gives it super-monkey abilities?

For a monkey it's impossible to design... pretty much anything beside a few simple tools. So, no. A monkey cannot design a bow, a loom, a tractor, a computer, or an AI of any kind.

We had designed many tools that beat us in various aspects. This is an invalid analogy.

Re: Will scaling work?

#129

Post by cavisne »

If the size of the internet is really a bottleneck it seems Google is in quite a strong position.

Assuming they have effectively a log of the internet, rather than counting the current state of the internet as usable data we should be thinking about the list of diffs that make up the internet.

Maybe this ends up like Millenium Management where a key differentiator is having access to deleted datasets.

Re: Will scaling work?

#130

Post by nextworddev »

I am in the believer camp for simple reasons: 1) we haven’t even scratched the surface of government led investments into AI, 2) AI itself could probably discover better architectures than transformers (willing to bet heavily on this)
Post reply on HN