Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s policies hinder reproducible research on language models

aisnakeoil.substack.com

Post reply on HN
Print view View original

101–110 of 394 posts

Re: OpenAI’s policies hinder reproducible research on language models

#101

Post by beambot »

skissane wrote: post #90
beambot wrote: post #81

Earlier quoted context omitted.

> it was firewalled and available only to large corporations and those with personal relations to big tech execs. The previous call-out to IBM seems relevant: before PCs, this exact statement would've been true for (mini)computers and mainframes.

> > it was firewalled and available only to large corporations and those with personal relations to big tech execs. > The previous call-out to IBM seems relevant: before PCs, this exact statement would've been true for (mini)computers and mainframes. Pre-PCs, IBM mainframes actually were quite open – up until the mid-1970s, IBM released its mainframe operating systems into the public domain . On the software side, th…

> The main way in which IBM mainframes in the 1950s-1970s were "firewalled" was simply by being fiendishly expensive – most people's houses cost significantly less.

This was the primary aspect I was referring to, in the same way that training a ChatGPT-like NN can be (or could become) prohibitively expensive.

But your comments about openness are relevant on an entirely different axis.

Re: OpenAI’s policies hinder reproducible research on language models

#103

Post by renonce »

danbmil99 wrote: post #83
atleastoptimal wrote: post #32

I understand any individual's company anti-competitive measures. OpenAI looks at Google the same way Apple looked at IBM in the 80s. What I'm worried about is a lot of the talk about guarding models, public safety and misuse of models will end up leading every big company to pull public access of their APIs. We might look at 2022-2023 as a brief golden age when regular people could use stuff like GPT-4 before it was…

Time to get serious about competitive open source models. Can't we do a seti at home sort of thing to distribute the training?

Training is a bandwidth-intensive operation and requires huge (20Gbps+ stable and uninterrupted) bandwidth between all peers.

Re: OpenAI’s policies hinder reproducible research on language models

#104

Post by gorbypark »

andrewmcwatters wrote: post #40

I've been busy with a number of projects and haven't had time to look into this but have been dying to know; has anyone recreated the architecture that OpenAI uses for text-davinci-003, InstructGPT, and ChatGPT that simply doesn't have training data? This is a reproducibility problem of its own sort. I mean, the papers are there out in the open if I understand correctly, but I don't know if anyone's actually built th…

I believe the best results will come from training the base LLM on as many sources of quality information as possible, and then fine tuning it with a narrower set of data later on. Here’s a small scale example where someone took LLaMa/Alpaca and fined tuned it with all the scripts from the first 12 seasons of The Simpsons. https://replicate.com/blog/fine-tune-llama-to-speak-like-hom...

Re: OpenAI’s policies hinder reproducible research on language models

#105

Post by redox99 »

thatsadude wrote: post #93
redox99 wrote: post #39

Earlier quoted context omitted.

Google (and others) may not have released model weights, but they've published papers, which is ultimately what makes the field advance. OpenAI not only did not publish any GPT4 paper, they haven't even said how many parameters it has.

Google published papers but has anybody be able to replicate their results?

Yes? Attention is all you need and Alpha Zero are the first that come to mind, but there are thousands.

Re: OpenAI’s policies hinder reproducible research on language models

#106

Post by menacingly »

Eji1700 wrote: post #43

Open AI has been doing sketchyish things long before Chat GPT, and I think it's something people are eventually going to notice more and more (then again people were swearing that Musk walked on water for waaaaaay too long given his actions so fuck if I know). They're 100% marketing FIRST. I don't think they'll outright lie, but they will absolutely screw with their data in such a way to make it look waaay more impre…

Dropping the tactical nuke of ChatGPT was PR brilliance, nearly anyone would kill for shifting the public conversation that dramatically. That kind of marketing first is a synonym for "winner", it almost doesn't matter what the actual product is, or if it works.

But it does, and then look at the impossibility of their position. If the massive cost of research and operations _augments a profitable line of business_, it is perhaps acceptable. Otherwise, you're just setting cash on fire.

Extremely difficult to operate as a non-profit, more realistic as a division than a standalone org, as much as I dislike saying my second pro-MS thing in a week, it makes sense, and I am OK with them operating anywhere except tucked inside an ad business.

Maybe this sort of thing should be operated by the government funding or whatever, but... it isn't.

Re: OpenAI’s policies hinder reproducible research on language models

#107

Post by godzillabrennus »

Eji1700 wrote: post #43

Open AI has been doing sketchyish things long before Chat GPT, and I think it's something people are eventually going to notice more and more (then again people were swearing that Musk walked on water for waaaaaay too long given his actions so fuck if I know). They're 100% marketing FIRST. I don't think they'll outright lie, but they will absolutely screw with their data in such a way to make it look waaay more impre…

This post will probably age as well as the guy who argued with Drew Houston on the market need for DropBox on here when he announced it.

Re: OpenAI’s policies hinder reproducible research on language models

#108

Post by mach1ne »

loveparade wrote: post #65
sva_ wrote: post #22

Since OpenAI didn't release the parameter count of GPT-4, I've been wondering/doubting if it is really much bigger than GPT-3. The release of GPT-3.5 has shown that they've found ways of drastically cutting down compute costs (an order of magnitude) while maintaining or even improving the quality of the model's outputs. Perhaps the reason that they didn't release the specifics of GPT-4 might be in part due to them wa…

Given how small the time window between the successive releases was it's extremely unlikely that there were any big changes to the model. Most likely it's just better preprocessed training data, more training data, trained for longer, performance optimizations for attention, or a few changes to layer sizes.

Your timeline is wrong, GPT-4 finished training already in August.

Re: OpenAI’s policies hinder reproducible research on language models

#109

Post by dclowd9901 »

atleastoptimal wrote: post #32

I understand any individual's company anti-competitive measures. OpenAI looks at Google the same way Apple looked at IBM in the 80s. What I'm worried about is a lot of the talk about guarding models, public safety and misuse of models will end up leading every big company to pull public access of their APIs. We might look at 2022-2023 as a brief golden age when regular people could use stuff like GPT-4 before it was…

Conversely, I also had a brief moment of panic considering a bunch people somehow bumbling their way into making actual factual general AI and causing the end of civilization.

I realize the cat’s out of the bag, but I feel like anything we can do to keep weaponized AI out of peoples hands as long as possible might not be the worst thing.

Re: OpenAI’s policies hinder reproducible research on language models

#110

Post by avereveard »

danbmil99 wrote: post #83
atleastoptimal wrote: post #32

I understand any individual's company anti-competitive measures. OpenAI looks at Google the same way Apple looked at IBM in the 80s. What I'm worried about is a lot of the talk about guarding models, public safety and misuse of models will end up leading every big company to pull public access of their APIs. We might look at 2022-2023 as a brief golden age when regular people could use stuff like GPT-4 before it was…

Time to get serious about competitive open source models. Can't we do a seti at home sort of thing to distribute the training?

"GPT-3 175B model required 3.14E23 flops" according to their marketing material. Seti at home was about 1PetaFlops iirc so about 3 years training, possibly less if you can generate enough attention to the project that the people with the beefy devices will partecipate.

The problem is that you need to train the full model you can't train aspect of it and even with each node doing independent tiny batches the network bandwidth for sinchronization would be massive.

Seti was massively parallelizable because of the nature of the job oddnt require sinchronization between every peer.

Post reply on HN