Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s policies hinder reproducible research on language models

aisnakeoil.substack.com

Post reply on HN
Print view View original

121–130 of 394 posts

Re: OpenAI’s policies hinder reproducible research on language models

#121

Post by whatshisface »

buildbot wrote: post #98
redox99 wrote: post #86

Earlier quoted context omitted.

> Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar. It's 99 pages of marketing material

Personally I disagree, there are lot of interesting tidbits in this paper. More than marketing would need at least.

What good bits did you find? (I'm not sure how fruitful the "OpenAI is a Microsoft department" debate is given that they are almost one and everybody knows it, but I am curious if anyone has found anything good in those many pages.)

Re: OpenAI’s policies hinder reproducible research on language models

#122

Post by mach1ne »

sillysaurusx wrote: post #60
loveparade wrote: post #48

What's most surprising to me is that OpenAI really seems to believe that not publishing details will save them from competition. Everyone knows how these models work, and while I'm sure there is a bunch of "secret sauce" that OpenAI has built for training and fine-tuning, it's ridiculous to believe that the research community and competitors like Google and Facebook can't figure out the same. They just haven't really…

Au contraire, no one knows how large GPT-4 is, which is the single best predictor of performance (for a model trained to convergence). The GPT-4 paper spent much of its time writing about this — they did some small scale experiments with 1/1000th the compute, then picked a loss level they wanted and trained GPT-4 till it got it. Neither the exact loss level nor the number of parameters are revealed by the paper. Unfo…

I think the big tech actors probably know. Information leaks and ultimately Google is spyware. Not that it will reach the public knowledge today, but that kind of information is difficult to keep in the bottle long time.

Re: OpenAI’s policies hinder reproducible research on language models

#123

Post by mach1ne »

redox99 wrote: post #39
fdgsdfogijq wrote: post #8

All these research science bureaucrats at Big Tech could have released LLM models or tried to develop what OpenAI did. But none of them did. We should applaud OpenAI for the innovation and let them do as they please.

Google (and others) may not have released model weights, but they've published papers, which is ultimately what makes the field advance. OpenAI not only did not publish any GPT4 paper, they haven't even said how many parameters it has.

Indeed Google came up with Transformers and decided to gift the model to humanity. By broad strokes it was luck that OpenAI chose the seemingly right path of AI.

Closest competitor DeepMind played games, which is intuitively closer to what humans do, but its relevance given aspects of deep learning is questionable.

Re: OpenAI’s policies hinder reproducible research on language models

#124

Post by cfn »

alden5 wrote: post #84
atleastoptimal wrote: post #32

I understand any individual's company anti-competitive measures. OpenAI looks at Google the same way Apple looked at IBM in the 80s. What I'm worried about is a lot of the talk about guarding models, public safety and misuse of models will end up leading every big company to pull public access of their APIs. We might look at 2022-2023 as a brief golden age when regular people could use stuff like GPT-4 before it was…

Just today I got Stanford's Alpaca-7b model running locally on my m1 mac, it’s just facebook’s Lamma-7b model which has been trained to complete tasks. It's getting close to the versatility of chatgpt where I could actually use it for everyday tasks. I don't think open source is that far away, especially considering how quickly Alpaca came out and how much better it is vs Lamma, which frequently would hallucinate and…

Can you link to instructions specific to the Mac. I can only find instructions for Alpaca with GPUs and PCs.

Re: OpenAI’s policies hinder reproducible research on language models

#126

Post by yipbub »

moi2388 wrote: post #116
jonathankoren wrote: post #50

Earlier quoted context omitted.

Competitors to Microsoft buy the self hosting github option.

Please name the competitors to Microsoft that use self hosted GitHub.

OS: linux,fedora, bsd Cloud: They're all closed-source.

Doesn't seem black and white since they have their hand in so many pies, but name a real competitor to Microsoft that uses github.com?

Re: OpenAI’s policies hinder reproducible research on language models

#127

Post by skissane »

UncleOxidant wrote: post #52
atleastoptimal wrote: post #32

I understand any individual's company anti-competitive measures. OpenAI looks at Google the same way Apple looked at IBM in the 80s. What I'm worried about is a lot of the talk about guarding models, public safety and misuse of models will end up leading every big company to pull public access of their APIs. We might look at 2022-2023 as a brief golden age when regular people could use stuff like GPT-4 before it was…

> We might look at 2022-2023 as a brief golden age when regular people could use stuff like GPT-4 Not sure about that since it seems to being baked into a lot of products at places like Microsoft. However, I'd change your statement a bit: We might look at 2023 as a brief golden age when regular people could access trained parameters (the LLaMA params) and run these models on their own machines (such as with alpaca.cp…

> However, I'd change your statement a bit: We might look at 2023 as a brief golden age when regular people could access trained parameters (the LLaMA params) and run these models on their own machines (such as with alpaca.cpp). I doubt we'll get access to LLM params again unless some kind of non-profit, actual open source organization is formed to produce them and put them out into the public domain.

There are a lot of people who'd love to have their own on-premise instance of ChatGPT (or equivalent), that they fully control, and could use for whatever purpose they want–even purposes that OpenAI might consider "harmful". They'd be happy to pay for that product if it were on offer.

Not just private individuals, even businesses – sending customer data to OpenAI involves lots of regulatory/legal/contractual hurdles, an on-premise offering avoids all those. Also, once you get to a certain scale, owning your own hardware works out cheaper than cloud.

If someone was to offer a ChatGPT-like service as an on-premise offering, I don't think they'd have any trouble finding people willing to pay for it. Even if they have to spend $X million to train a new model from scratch, I'm sure some VC would view it as a worthwhile investment. Of course, a free open source model would be even better, but a paid/commercial/proprietary on-premise model would remove many of the disadvantages of OpenAI. I'm 100% sure it is coming soon, I bet there are multiple teams working on it even as I type this.

Re: OpenAI’s policies hinder reproducible research on language models

#129

Post by buildbot »

whatshisface wrote: post #121
buildbot wrote: post #98

Earlier quoted context omitted.

Personally I disagree, there are lot of interesting tidbits in this paper. More than marketing would need at least.

What good bits did you find? (I'm not sure how fruitful the "OpenAI is a Microsoft department" debate is given that they are almost one and everybody knows it, but I am curious if anyone has found anything good in those many pages.)

I think the most interesting thing is the their ability to predict performance from loss and on a wide range of tasks using a much smaller model - this lets them fine tune their architecture and hypers, then run a single large training run to get full scale gpt4 - from the paper it sounds like they only trained the large model once, then did a Reinforcement learning with human feedback finetune.

Disclaimer - I work at Microsoft, in AI, and have no internal knowledge about gpt4.

Re: OpenAI’s policies hinder reproducible research on language models

#130

Post by atleastoptimal »

dclowd9901 wrote: post #109
atleastoptimal wrote: post #32

I understand any individual's company anti-competitive measures. OpenAI looks at Google the same way Apple looked at IBM in the 80s. What I'm worried about is a lot of the talk about guarding models, public safety and misuse of models will end up leading every big company to pull public access of their APIs. We might look at 2022-2023 as a brief golden age when regular people could use stuff like GPT-4 before it was…

Conversely, I also had a brief moment of panic considering a bunch people somehow bumbling their way into making actual factual general AI and causing the end of civilization. I realize the cat’s out of the bag, but I feel like anything we can do to keep weaponized AI out of peoples hands as long as possible might not be the worst thing.

That's true. The real question is whether or not the people in this new era calling the shots have the necessary capability and intentions to maximize the benefit these models will provide humanity and minimize the risk.
Post reply on HN