Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s policies hinder reproducible research on language models

aisnakeoil.substack.com

Post reply on HN
Print view View original

81–90 of 394 posts

Re: OpenAI’s policies hinder reproducible research on language models

#81

Post by beambot »

atleastoptimal wrote: post #32

I understand any individual's company anti-competitive measures. OpenAI looks at Google the same way Apple looked at IBM in the 80s. What I'm worried about is a lot of the talk about guarding models, public safety and misuse of models will end up leading every big company to pull public access of their APIs. We might look at 2022-2023 as a brief golden age when regular people could use stuff like GPT-4 before it was…

> it was firewalled and available only to large corporations and those with personal relations to big tech execs.

The previous call-out to IBM seems relevant: before PCs, this exact statement would've been true for (mini)computers and mainframes.

Re: OpenAI’s policies hinder reproducible research on language models

#82

Post by buildbot »

redox99 wrote: post #39
fdgsdfogijq wrote: post #8

All these research science bureaucrats at Big Tech could have released LLM models or tried to develop what OpenAI did. But none of them did. We should applaud OpenAI for the innovation and let them do as they please.

Google (and others) may not have released model weights, but they've published papers, which is ultimately what makes the field advance. OpenAI not only did not publish any GPT4 paper, they haven't even said how many parameters it has.

Then what is this? 99 pages of bullshit? https://arxiv.org/pdf/2303.08774.pdf

Re: OpenAI’s policies hinder reproducible research on language models

#83

Post by danbmil99 »

atleastoptimal wrote: post #32

I understand any individual's company anti-competitive measures. OpenAI looks at Google the same way Apple looked at IBM in the 80s. What I'm worried about is a lot of the talk about guarding models, public safety and misuse of models will end up leading every big company to pull public access of their APIs. We might look at 2022-2023 as a brief golden age when regular people could use stuff like GPT-4 before it was…

Time to get serious about competitive open source models. Can't we do a seti at home sort of thing to distribute the training?

Re: OpenAI’s policies hinder reproducible research on language models

#84

Post by alden5 »

atleastoptimal wrote: post #32

I understand any individual's company anti-competitive measures. OpenAI looks at Google the same way Apple looked at IBM in the 80s. What I'm worried about is a lot of the talk about guarding models, public safety and misuse of models will end up leading every big company to pull public access of their APIs. We might look at 2022-2023 as a brief golden age when regular people could use stuff like GPT-4 before it was…

Just today I got Stanford's Alpaca-7b model running locally on my m1 mac, it’s just facebook’s Lamma-7b model which has been trained to complete tasks. It's getting close to the versatility of chatgpt where I could actually use it for everyday tasks. I don't think open source is that far away, especially considering how quickly Alpaca came out and how much better it is vs Lamma, which frequently would hallucinate and often didn't make sense.

Example of prompt to Lamma-7b:

  > write a poem about open ai not being open source
  Open AI is not really “open”
  As this project isn’t open sore
  It can be seen as closed fortress,
  Inside which secrets are hidden.
Not mind blowing but still really interesting, I will note that its much better at things like answering trivia where's there's already lots of examples in its model

Re: OpenAI’s policies hinder reproducible research on language models

#85

Post by mcaledonensis »

ar9av wrote: post #75
sva_ wrote: post #22

Since OpenAI didn't release the parameter count of GPT-4, I've been wondering/doubting if it is really much bigger than GPT-3. The release of GPT-3.5 has shown that they've found ways of drastically cutting down compute costs (an order of magnitude) while maintaining or even improving the quality of the model's outputs. Perhaps the reason that they didn't release the specifics of GPT-4 might be in part due to them wa…

I saw this coming a long time ago and I'm still very pissed off. For three reasons: 1. We are all forced to use the damn "chat" API instead of regular completions. Can't wait to have to deal with chatgpt's conversations in order to get a few lines of code out 2. We loose the super valuable 'insert' and 'edit' modes, which were great for code 3. 3-day notice period? that's going to be a hell for people who are actuall…

Completion API for GPT-4 will be there soon. With extra stop tokens, but better than nothing. A compromise.

And it's not like what OpenAI did was an impossible magic trick. They've had a right team composition. And three insights. All present in the literature. Repeat that, you'll have GPT-4. But GPT-5. Well, that one is different game.

As to being open, they are still relatively open. Consider Apple, for example. No one complains about Apple being a bit skittish. Well, OpenAI got a bit skittish too. It's a period. They'll stabilize. And their setup of the company, with the non-profit board in control, profit caps is a really interesting try at the corporate design.

Re: OpenAI’s policies hinder reproducible research on language models

#86

Post by redox99 »

buildbot wrote: post #82
redox99 wrote: post #39

Earlier quoted context omitted.

Google (and others) may not have released model weights, but they've published papers, which is ultimately what makes the field advance. OpenAI not only did not publish any GPT4 paper, they haven't even said how many parameters it has.

Then what is this? 99 pages of bullshit? https://arxiv.org/pdf/2303.08774.pdf

> Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.

It's 99 pages of marketing material

Re: OpenAI’s policies hinder reproducible research on language models

#87

Post by nl »

numberalltheway wrote: post #44

It's even more frustrating that, from what I can tell, there is nothing published about how GPT-4 improved. I take specific exception to the hiding of the data and techniques used to generate the model. There must be something specific going on in the model that is allowing it to perform better than GPT-3 and better than what any contemporaries are able to produce. Not publishing this information hinders the further…

> It's even more frustrating that, from what I can tell, there is nothing published about how GPT-4 improved.

There's the GPT-4 Technical Report which gives benchmark results vs GPT-3.5, PaLM, Chinchilla, LLAMA and other models depending on the benchmark.

https://cdn.openai.com/papers/gpt-4.pdf

Re: OpenAI’s policies hinder reproducible research on language models

#88

Post by blueorange8 »

sva_ wrote: post #64
Eji1700 wrote: post #43

Open AI has been doing sketchyish things long before Chat GPT, and I think it's something people are eventually going to notice more and more (then again people were swearing that Musk walked on water for waaaaaay too long given his actions so fuck if I know). They're 100% marketing FIRST. I don't think they'll outright lie, but they will absolutely screw with their data in such a way to make it look waaay more impre…

My comment might've seemed like I judge them for trying to make a profit - I don't, since there's nothing wrong with that. I was more pointing to the fact that they probably need to make a profit, rather sooner than later, so they aren't shackled by M$ and can be an independent company.

If they ever do become an independent company you can be sure that Microsoft would already have sucked them dry. Microsoft will never let them go now as long as they are valuable.

Re: OpenAI’s policies hinder reproducible research on language models

#89

Post by gentoo »

raincole wrote: post #76
Eji1700 wrote: post #43

Open AI has been doing sketchyish things long before Chat GPT, and I think it's something people are eventually going to notice more and more (then again people were swearing that Musk walked on water for waaaaaay too long given his actions so fuck if I know). They're 100% marketing FIRST. I don't think they'll outright lie, but they will absolutely screw with their data in such a way to make it look waaay more impre…

If they're 100% marketing first, and still made the most impressive AI product so far, you really need to question what all the other companies are doing. (before someone says Google or Meta's models are bigger or something... I mean product, not models)

openAI is in the business of releasing impressive tech demos, Google is in the business of providing search results. I would believe that Google is further along towards creating something useful, but they still don't have anything that's better than their existing search product.

Re: OpenAI’s policies hinder reproducible research on language models

#90

Post by skissane »

beambot wrote: post #81
atleastoptimal wrote: post #32

I understand any individual's company anti-competitive measures. OpenAI looks at Google the same way Apple looked at IBM in the 80s. What I'm worried about is a lot of the talk about guarding models, public safety and misuse of models will end up leading every big company to pull public access of their APIs. We might look at 2022-2023 as a brief golden age when regular people could use stuff like GPT-4 before it was…

> it was firewalled and available only to large corporations and those with personal relations to big tech execs. The previous call-out to IBM seems relevant: before PCs, this exact statement would've been true for (mini)computers and mainframes.

> > it was firewalled and available only to large corporations and those with personal relations to big tech execs.

> The previous call-out to IBM seems relevant: before PCs, this exact statement would've been true for (mini)computers and mainframes.

Pre-PCs, IBM mainframes actually were quite open – up until the mid-1970s, IBM released its mainframe operating systems into the public domain. On the software side, the IBM S/360 was actually a lot more open than the IBM PC was – OS/360 was public domain with publicly available source code and even design documents (logic manuals), PC-DOS was copyrighted proprietary software whose source code and design documents were only publicly released decades after it had ceased to be commercially relevant.

As we move through the 1970s, IBM became less and less open. The core OS remained in the public domain, but new features were increasingly only available in copyrighted add-ons – but IBM still shipped its customers source code, design documents, etc, for those add-ons. Finally, in 1983, IBM announced that the public domain core was being replaced by a new copyrighted version, for which it would withhold source code access from customers ("object code only", or "OCO" for short).

The main way in which IBM mainframes in the 1950s-1970s were "firewalled" was simply by being fiendishly expensive – most people's houses cost significantly less.

It is true that IBM did engage in anti-competitive business practices, but those were primarily non-technological in nature – contractual terms, pricing, etc – the kind of techniques which Thomas J. Watson Sr had mastered as an NCR sales executive in the lead-up to World War I. In fact, a big contributor to IBM becoming "less open" was the US Justice Department's 1969 anti-trust lawsuit, which led to IBM unbundling software and services from hardware–and its software culture became progressively more closed as software came to be seen as a product in its own right.

Post reply on HN