Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will scaling work?

dwarkeshpatel.com

Post reply on HN
Print view View original

221–230 of 289 posts

Re: Will scaling work?

#221

Post by berniedurfee »

jeremyjh wrote: post #196
berniedurfee wrote: post #188

Earlier quoted context omitted.

Lol, at least then your comment wouldn’t have bothered me so much!

I'm sorry I hurt your feelings, it wasn't my intention. For what its worth, I actually think there is a good chance that you are right - that there is something missing in LLMs that still won't be present in bigger LLMs. I mostly meant that an LLM would be more organized around the source material and address specific points. I actually asked ChatGPT 4 to do so, and it produced the sort of reasonable but unremarkable…

Ah, gotcha, no worries, thanks for the reply!

Re: Will scaling work?

#222

Post by hardwaregeek »

Jensson wrote: post #141
hardwaregeek wrote: post #126

Earlier quoted context omitted.

It's important to remember that the internet is still very very new. Like the generation of digital natives are barely in adulthood. Sure, it's existed in some form for about 40 years, but most of the world didn't have access for the longest time. I wouldn't be surprised if we see massive changes in the next 20 years from the people who grew up on the web (specifically people outside the United States and Europe, whe…

"Digital native" are the people who grew up with computers. Many kids born in 1980's and later grew up with computers in their earliest memories. I'd call the current generation "Social media natives", because that is the biggest difference from the previous generation. 90s kids grew up with games and communication, but they were free from facebook, youtube and instagram.

Many kids in the US and Europe grew up with computers[1]. There's a large large population of people from Africa, South America, Asia, etc. who just got internet access in the last 10-20 years.

[1]: Or came from wealthy families elsewhere.

Re: Will scaling work?

#223

Post by bee_rider »

laurencerowe wrote: post #206
Negitivefrags wrote: post #166

Earlier quoted context omitted.

I remember long ago reading an argument that information technology has not actually increased productivity. I really wish I could find a source for this now, but I just can't seem to find it anywhere on the internet. Here it is anyway: The administration of the Tax Service uses 4% of the total tax revenue it generates. This percentage has stayed relatively fixed over time. If IT really improved productivity, wouldn'…

> The administration of the Tax Service uses 4% of the total tax revenue it generates. This percentage has stayed relatively fixed over time. The tax administration is far more efficient than that. The IRS has 79K workers out of a total workforce of 158M, or 1/2000 workers. Federal taxes are about 19% GDP (28% of GDP including state and local taxes.) The IRS costs $14.3B to run and collects 19% of $25.46T = $4,800B o…

The idea of measuring the “efficiency” of the tax system in terms of money in per money out seems a bit odd to me in the first place. Taxes don’t create wealth, the job is to destroy it at the correct rate.

Don’t get me wrong, I’m not a “taxes are theft” dummy or anything like that. Taxes are an important knob in shaping the economy. But a better functioning tax collection agency should more effectively implement the rules of whose money is collected for deletion, not just collect more money generally.

Re: Will scaling work?

#224

Post by Xelynega »

lnxg33k1 wrote: post #183
slibhb wrote: post #49

The best analogy for LLMs (up to and including AGI) is the internet + google search. Imagine explaining the internet/google to someone in 1950. That person might say "Oh my god, everything will change! Instantaneous, cheap communication! The world's information available at light speed! Science will accelerate, productivity will explode!" And yet, 70 years later, things have certainly changed, but we're living in the…

Imagine telling those same people in the 50s that all those changes in productivity would come for the benefit of no one since the work week would be the same and purchasing power would decline

Don't have to imagine, same thing is happening right now with LLMs.

I see "AI safety" brought up as a laughable attempt at stopping the progress of LLMs, when in reality the people talking about "AI safety" are the people trying to say that the majority will not benefit from this technology.

Re: Will scaling work?

#225

Post by gradschoolfail »

bee_rider wrote: post #60
slibhb wrote: post #49

The best analogy for LLMs (up to and including AGI) is the internet + google search. Imagine explaining the internet/google to someone in 1950. That person might say "Oh my god, everything will change! Instantaneous, cheap communication! The world's information available at light speed! Science will accelerate, productivity will explode!" And yet, 70 years later, things have certainly changed, but we're living in the…

The internet did change things pretty dramatically. Productivity at information communication tasks just isn’t the entire economy. I think we are massively more productive. Some of the biggest new companies are ad companies (Google, Facebook), or spend a ton of their time designing devices that can’t be modified by their users (Apple, Microsoft). Even old fashioned companies like tractor and train companies have time…

Going slightly beyond armchair economics, here are a couple of articles which discuss the lack of evidence for internet-based productivity so far:

https://archive.ph/baneA https://archive.ph/TrHYN

“Our central theme is that computers and the Internet do not measure up to the Great Inventions of the late nineteenth and early twentieth century, and in this do not merit the label of Industrial Revolution,”

— Robert Gordon, actual economist

Re: Will scaling work?

#226

Post by elktown »

lern_too_spel wrote: post #219
elktown wrote: post #218

Earlier quoted context omitted.

No, not at all. The only thing I did was to react on how ridiculous that oversimplification is and how such a thing can only come about due to an embarrassing amount of hubris currently going around our field with relation to "AI". It's a hand-wavy "Eh, how hard can it be?" comment to rationalize ML being a pathway to AGI.

There is nothing oversimplified in that description at all. Here is an equivalent description from a neuroscience textbook: "Neuroscience is the study of the nervous system, the collection of nerve cells that interpret all sorts of information which allows the body to coordinate activity in response to the environment." https://openbooks.lib.msu.edu/introneuroscience1/chapter/wha... This is even simpler than the RL a…

Yet again you just pick out a word, "oversimplified" in this case, ignores the rest of the reply - and more importantly, the entire context of all these replies - and goes on to make a nonsensical comparison with the ever-present one-liner that all textbooks of all academic disciplines have.

Have you not understood by now that my critic of our field's AI-hubris is a general one - and thus not hinged on exact wordings? I could've written similar hubris related replies on multiple other comments in this thread. It's my reaction. It's my exasperation.

Re: Will scaling work?

#227

Post by Xelynega »

gitfan86 wrote: post #6
hokeone wrote: post #4

>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…

Over the past year there have been advances in making models smaller while keeping performance high. So if that continues then he is wrong unless he is defining LLMs in a strict way that does not include new improvement in the future

It's not about the size of the models, it's about the size of the training data.

Humans are able to begin to generalize with a single persons experiences over less than a year, so the fact that LLMs cannot with billions of person-years of information could be an indicator of their inability to generalize no matter how much training data you throw at it.

Re: Will scaling work?

#228

Post by mk_stjames »

sgt101 wrote: post #123

>Here’s one of the many astounding finds in Microsoft Research’s Sparks of AGI paper. They found that GPT-4 could write the LaTex code to draw a unicorn. a lot of people have tried to replicate this, I have tried. It's very hard to get GPT-4 to draw a unicorn, also asking it to draw an upside down unicorn is even harder.

Ask it to draw an image of a unicorn in SVG format instead.

I've did this and got a result. Asked for a cat wearing a hat. It drew a circle with dots as eyes and sorta-whiskers with lines and a triangle for the hat. All using just SVG vector code.

The claim is result shows that there is understanding of not just words about cats and hats and connections to shapes but also a bit of spatial awareness in the x,y coords needed on the SVG canvas to place the shapes.

I was able to do with with several different little characters / cartoons in SVG and while it completely fails every now and then it was better than I would have thought.

I think it has been exposed to SVG (via the web) way more than LaTeX vector drawings.

Re: Will scaling work?

#229

Post by Xelynega »

midlightdenight wrote: post #143
sgt101 wrote: post #123

>Here’s one of the many astounding finds in Microsoft Research’s Sparks of AGI paper. They found that GPT-4 could write the LaTex code to draw a unicorn. a lot of people have tried to replicate this, I have tried. It's very hard to get GPT-4 to draw a unicorn, also asking it to draw an upside down unicorn is even harder.

The model of GPT-4 those researchers had was not the same that’s available to the public. It’s assumed it was far more capable before alignment training (or whatever it’s called).

That's convenient. Typically one of the markers of good science is reproducability. How can we trust any of the information coming out of these studies if it can't be reproduced?

Re: Will scaling work?

#230

Post by sbierwagen »

Eddy_Viscosity2 wrote: post #20
auggierose wrote: post #14

Earlier quoted context omitted.

And yet, we reached the moon, and I would say airplanes were a necessary step on the way, even if only for psychological reasons. For airplanes we had at least an example in nature, birds. But I am not aware of any animal that travelled from earth to the moon on its own, except us.

But we didn't use airplanes to get there. It needed a new approach, different propulsion, different fuel, different attitude control, etc. etc. LLM may be a necessary step to get to AGI, but it (probably) won't be the one that achieves that goal.

I mean, if you go through the Apollo program contractors it's a who's who of aerospace. Boeing, North American Aviation, Grumman Aircraft, McDonnell Aircraft Corporation, Bell Aerosystems, Rocketdyne...

Electrical parts ran at aviation-standard 400hz. Aviation gyroscopes and aviation instruments. Structural parts made of aviation aluminum alloys. Astronauts that are all airplane test pilots. I can imagine doing Apollo from complete scratch (using car manufacturers that have to invent aluminum-handling tech starting from nothing) but it would have taken a lot more than the decade Apollo took.

Post reply on HN