Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training for one trillion parameter model backed by Intel and US govt has begun

techradar.com

Post reply on HN
Print view View original

11–20 of 267 posts

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#11

Post by vouaobrasil »

danielbln wrote: post #8
vouaobrasil wrote: post #3

Not surprising. Despite the enormous energy costs and the threat to humanity by creating technology that we can't control, governments and corporations will build bigger and more sophisticated models just because they have to do so to compete. It is the prisoner's dilemma: we end up with a super-advanced AI that will disrupt and make society worse because entities are competing where the metric of success is short-te…

It might be dangerous, but how is it useless? There is also an enormous positive gain to be had in terms of autonomous discovery, science progress and so on. And prisoner's dilemma means "we" will not stop pursuing this, even of there is a net negative benefit to humanity overall.

Yes, I believe there is a net negative benefit.

I do not think there is an enormous positive gain. Science progress in my opinion has gone too far in many respects and we should not be able to generate science so easily because we will not have the wisdom to use it. Although I love science, it's use is far too lackadaisical now -- see for example the climate crisis and efficient resource extraction of fossil fuels.

Then we can imagine things like life-extending drugs, immunity from all disease (bad because it implies overpopulation), methods to generate oxygen (bad because it will mean we won't value the natural ecosystem as much), etc.

Smartphones, computer technology (human-isolating and community-destroying), etc. Pretty much all modern inventions are making life worse.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#12

Post by huijzer »

bradley13 wrote: post #6

The solution won't be just "bigger". A model with a trillion parameters will be more expensive to train and to run, but is unlikely to be better. Think of the early days of flight, you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things. Improving AI will involve architectural changes. No human requires the amount of training data we…

Karpathy in his recent video [1] agrees, but at this point scaling is a very reliable way to better accuracy.

[1]: https://youtu.be/zjkBMFhNj_g?si=eCH04466rmgBkHDA

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#13

Post by uticus »

vouaobrasil wrote: post #3

Not surprising. Despite the enormous energy costs and the threat to humanity by creating technology that we can't control, governments and corporations will build bigger and more sophisticated models just because they have to do so to compete. It is the prisoner's dilemma: we end up with a super-advanced AI that will disrupt and make society worse because entities are competing where the metric of success is short-te…

"First rule in government spending: why build one when you can have two at twice the price?" - Contact

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#14

Post by omneity »

vouaobrasil wrote: post #3

Not surprising. Despite the enormous energy costs and the threat to humanity by creating technology that we can't control, governments and corporations will build bigger and more sophisticated models just because they have to do so to compete. It is the prisoner's dilemma: we end up with a super-advanced AI that will disrupt and make society worse because entities are competing where the metric of success is short-te…

What makes you think this technology is out of control? If anything a 1T param model would only run in like 5-10 computers in this world. As much control as it gets.

By the way this is doomer rhetoric. Bashing scientific advancements as dangerous or useless by trying to attach vacuously scary ideas that are as speculative as they are outlandish.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#15

Post by omneity »

danpalmer wrote: post #7
Toolbox1337 wrote: post #4

Not bad for a start by the US government. A bit more than half as powerful as the 1.7 trillion parameter GPT4 model.

A bit more than half the size, it remains to be seen how powerful it is. There's clearly a non-linear relationship to model size, and it's also clear that it's hard to assess the power of these models anyway.

GPT-4 is unlikely to be 1.7T params. This is a number floating around in the internet with no justification.

The largest US open model is Google’s Switch-C which is 1.6T and only because it is a Mixture of experts model, i.e. it is constituted of many small models working together.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#17

Post by vouaobrasil »

omneity wrote: post #14
vouaobrasil wrote: post #3

Not surprising. Despite the enormous energy costs and the threat to humanity by creating technology that we can't control, governments and corporations will build bigger and more sophisticated models just because they have to do so to compete. It is the prisoner's dilemma: we end up with a super-advanced AI that will disrupt and make society worse because entities are competing where the metric of success is short-te…

What makes you think this technology is out of control? If anything a 1T param model would only run in like 5-10 computers in this world. As much control as it gets. By the way this is doomer rhetoric. Bashing scientific advancements as dangerous or useless by trying to attach vacuously scary ideas that are as speculative as they are outlandish.

It's hardly vacuous. What about these scientific advancements?

1. Fossil fuel extraction -- we are killing our planet.

2. Plastic chemistry -- there are 5 trillion pieces of plastic in the ocean killing seabirds, turtles, fish (https://doi.org/10.1371/journal.pone.0111913)

3. Concrete engineering -- building cities that move people away from a connection with nature

4. Veterinary medicine -- supporting global meat industry that is akin to torture on animals

5. Vehicle engineering -- capable of making vehicles that can kill MILLIONS of square meters of forest.

Our technology allows us to live comfortable lives but ONLY at the EXPENSE of millions of nonhuman animals (wild and domestic) that are KILLED every single day for our comforts. We dish out untold violence every second for our advancements.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#18

Post by barnabee »

vouaobrasil wrote: post #11
danielbln wrote: post #8

Earlier quoted context omitted.

It might be dangerous, but how is it useless? There is also an enormous positive gain to be had in terms of autonomous discovery, science progress and so on. And prisoner's dilemma means "we" will not stop pursuing this, even of there is a net negative benefit to humanity overall.

Yes, I believe there is a net negative benefit. I do not think there is an enormous positive gain. Science progress in my opinion has gone too far in many respects and we should not be able to generate science so easily because we will not have the wisdom to use it. Although I love science, it's use is far too lackadaisical now -- see for example the climate crisis and efficient resource extraction of fossil fuels. T…

This reads like a combination of pessimism and nostalgia has allowed you to cherry pick the worst of now and the best of the past and conince yourself it is net bad.

I see all of these negatives, and agree it is clear there is much that is bad in the world (there always has been) and much to be improved (ditto), including new problems created by science, technology, and human greed & selfishness. But I also see many positives and benefits, I see many problems that used to exist and don't anymore, and I see much evidence that we continue on average, to make progress. When I look to the past I do not wish I lived there.

Science have many times saved my friends and families' lives or enabled them to live more fully after an accident or sickness

Science made climate change possible but greed, raw capitalism, etc. put us on the path there. Science makes it possible to find a way out, slow or even reverse it eventually, and mitigate the worst results.

Populations are shrinking in developed countries despite people living longer.

Computers and the internet created and connected many new communities, too. The negative effects are again often the result of unbridled capitalism. Young people especially are fighting back against the negative effects and taking control, building technology to manage those hamrful effects.

etc…

I believe both that:

(a) it is not possible to stop humans acquiring and using knowledge, advancing science, developing technology, and evolving culturally and socially; and

(b) we shouldn't want to, even though it means we will create new problems and lose (or consign to a smaller "museum exhibit" role) some aspects of the culture and societies we have today, because on balance we will solve more and bigger problems than we create and build better and more interesting cultures and societies than those of the past.

It's not without existential risks, but those are overstated or analysed pseudoscientifically at best, by many of those (and certainly the loudest) who worry about them.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#19

Post by py4 »

huijzer wrote: post #12
bradley13 wrote: post #6

The solution won't be just "bigger". A model with a trillion parameters will be more expensive to train and to run, but is unlikely to be better. Think of the early days of flight, you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things. Improving AI will involve architectural changes. No human requires the amount of training data we…

Karpathy in his recent video [1] agrees, but at this point scaling is a very reliable way to better accuracy. [1]: https://youtu.be/zjkBMFhNj_g?si=eCH04466rmgBkHDA

This. We have not exhausted all the techniques at our disposal yet. We do need to look for a new architecture though, but these are orthogonal

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#20

Post by skerit »

vouaobrasil wrote: post #3

Not surprising. Despite the enormous energy costs and the threat to humanity by creating technology that we can't control, governments and corporations will build bigger and more sophisticated models just because they have to do so to compete. It is the prisoner's dilemma: we end up with a super-advanced AI that will disrupt and make society worse because entities are competing where the metric of success is short-te…

Your profile says "I am not rabidly against technology, but I promote a BALANCED, severely critical look at it"

"Humanity should give up this useless development of AI"

So balanced.

Post reply on HN