Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 details leaked?

threadreaderapp.com

Post reply on HN
Print view View original

161–170 of 648 posts

Re: GPT-4 details leaked?

#161

Post by PostOnce »

charcircuit wrote: post #117
PostOnce wrote: post #19

"Open" AI, a charity to benefit us all by pushing and publishing the frontier of scientific knowledge. Nevermind, fuckers, actually it's just to take your jobs and make a few VCs richer. We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us. https://github.com/ggerganov/llama.cpp https://github.com/openlm-research/open_llama https://huggingface.co/TheBlok…

>There are no license issues like with the facebook llama. OpenLLaMa uses a dataset which does not seem to have gotten propper commercial licensing for the training data. There is potential licensing issues because the copyright situation is not well defended.

So far as anyone knows, this is not a derivative work, its transformative, and therefore not subject to any licensing requirement.

You're right though, that's arguably still up for debate, but I think the precedent of transformative work is pretty well attested.

Re: GPT-4 details leaked?

#162

Post by astrange »

holoduke wrote: post #156
behnamoh wrote: post #27

Earlier quoted context omitted.

Google is jokingly behind in terms of LLMs. They've done a pretty good job at incorporating vision and audio ML models into their ecosystem, but they underestimated language.

Their translation service is based on llms and is commercially a successful product.

Transformer models rather than LLMs surely. ChatGPT behaves nothing like Google Translate.

Re: GPT-4 details leaked?

#163

Post by moconnor »

RC_ITR wrote: post #20

For all the 'I know every number' certainty of this post, there's some weird stuff: >(Today, the pre-training could be done with ~8,192 H100 in ~55 days for $21.5 million at $2 per H100 hour.) Why flex both system size and training time to arbitrary numbers? >For example, MoE is incredibly difficult to deal with on inference because not every part of the model is utilized on every token generation. This means parts m…

No, the post is correct about MQA. A KV-cache only caches the key and value heads. The point of MQA is that your KV-cache is 1/heads smaller than usual because of this sharing.

Having multiple query heads does not affect the cache size, which is the limiting factor in MHA decoding for both memory capacity and bandwidth reasons.

Re: GPT-4 details leaked?

#164

Post by holoduke »

getmeinrn wrote: post #47

>If their cost in the cloud was about $1 per A100 hour, the training costs for this run alone would be about $63 million. If someone legitimate put together a crowd funding effort, I would donate a non-insignificant amount to train an open model. Has it been tried before?

Some kind of SETI project, but for training a high number parameter llm would be awesome.

Re: GPT-4 details leaked?

#165

Post by RC_ITR »

flir wrote: post #121
downWidOutaFite wrote: post #85

Earlier quoted context omitted.

The "regulatory capture" conspiracy theory makes no sense to me. It takes 9 figures in cash to create and run one of these super big models. Only big tech was ever going to create them, and big tech is already very experienced at navigating regulation, regulation wasn't ever going to stop them from competing. And in general, our democracy works better than the nihilist libertarians give it credit for.

I bet it'll be 6 figures within 18 months.

The thing about these models is compute scales quadratically with model dimensionality and memory scales quadratically with sequence length.

We are nowhere near diminishing returns for either variable, so sure current models maybe scale quickly but the cutting edge will want as much compute as possible for a long time.

That’s kind of the humor of everyone saying this leak somehow leaves OpenAI vulnerable.

The work isn’t deciding if an MoE is the right architecture or not, it’s how to run 25k GPUs concurrently in a fault tolerant way (likely the true reason for the deep Azure links).

Re: GPT-4 details leaked?

#166

Post by astrange »

H8crilA wrote: post #107
swyx wrote: post #90

Earlier quoted context omitted.

as a fun ancedote, the Google Bard's implicit code execution update from *last month*, advertised by Sundar... no longer works https://twitter.com/swyx/status/1678495067663925248 i'd love to know whats going on in that team.

Probably safety-driven terror. They really really want to get their bots going, but in every single meeting some PM or other concerned engineer talks about safety and f**s up the entire meeting. They even made the bot not respond to arithmetics questions because the bot is bad at this, lol. Someone who knows how to modify the bot had actually spent their time on something as unimportant as that.

Bard being bad at anything else doesn't seem to stop it. It hallucinates at the drop of the hat. Asking it almost any question implying X nonexistent thing exists causes it to make that thing up.

Re: GPT-4 details leaked?

#167

Post by jona-f »

wiz21c wrote: post #148
kyledrake wrote: post #65

Earlier quoted context omitted.

Unfortunately I've found the current OSS models to be vastly inferior to the OpenAI models. Would love to see someone actually get close to what they can do with GPT-3.5/4, except capable of running on commodity GPUs. What's the most impressive open model so far?

What are all the researchers in universities doing ? Couldn't they improve these models (they do have big brains after all) with tax payer's money and put the results under some cool open source license...

Yes, they are doing the improving, but then you need loads of money to do the learning no university can afford. So now big tech is hiring promising university researchers for good money to scale up their research. This could be solved by massive decentralization where millions of users provide compute with their gpus and i think it will be at some point, cause i believe foss is more powerful than this openai bs. There are people working on this, but afaik the techniques aren't quite there. You need a different kind of model with much more parallelization then what is currently used.

Re: GPT-4 details leaked?

#168

Post by Al0neStar »

mahathu wrote: post #134
KronisLV wrote: post #112

Earlier quoted context omitted.

Huh, more power to those folks then, I guess. But I can easily imagine more conventional forms of entertainment, as well. Like a game of D&D that's narrated by the AI, or a text based adventure set in the Mass Effect universe, Lord of the Rings, Warhammer or any other fandom, really. Maybe like those old Choose Your Own Adventure games. I think some companies are also experimenting with characters in video games that…

The application in games I'm most excited about is commenters in FIFA career mode that don't have a limited set of prerecorded voice lines, and take your recent games, formation changes etc into account too, like real commentators would. The recent installments already do that to a small degree. Of course this would also easily open the doors to having multiple commentators/analysts to choose from, each with their in…

Maybe for side quests but even that would be a debug hell.

Re: GPT-4 details leaked?

#170

Post by theLiminator »

xeckr wrote: post #135

If this is true, then: 1. Training took 21 yottaflops. When was the last time you saw the yotta- prefix for anything? 2. The training cost of GPT-4 is now only 1/3 of what it was about a year ago. It is absolutely staggering how quickly the price of training an LLM is dropping, which is great news for open source. The google memo was right about the lack of a moat.

The real moat is an abundance of high quality data.
Post reply on HN