Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s policies hinder reproducible research on language models

aisnakeoil.substack.com

Post reply on HN
Print view View original

61–70 of 394 posts

Re: OpenAI’s policies hinder reproducible research on language models

#61

Post by mvuksano »

randomwalker wrote: post #27
mvuksano wrote: post #14

I'm confused why people expect this stuff to be free? I'm surprised OpenAI was so open about their research so far. I don't blame them at all for not publishing the information. This stuff costs real money.

We don't expect it to be free -- please read the article. That's not the issue at all. It's like if you subscribe to a product that you need to do your job, and one day the company tells you that the product is going away in three days and that you need to switch to a different product (that isn't at all the same for your use case).

I don't think it's a smart idea to build any serious business using a tech that you can't replace. ChatGPT is great tool to help with coding for example but it's by no means substitute for an engineer. If someone starts a business by hiring a number of bootcampers and giving them ChatGPT hoping to run a serious business that way - well it's their risk to take... But no crying later...

Re: OpenAI’s policies hinder reproducible research on language models

#62

Post by aliston »

cardosof wrote: post #11

IMO established companies (Meta, Google, etc) had their researchers publish papers as a competitive benefit or way to attract talent from academia (a researcher wouldn't want to stop publishing). Companies didn't see an issue with doing that because those papers were not "giving away" the core of the company, for example, Facebook's DeepFace paper from 2014 couldn't hurt its ad business. OpenAI on the other hand will…

It will be really interesting to see if Google, Facebook etc. become more closed as a result. There was already a lot made of the fact that OpenAI hired away a group of engineers from DeepMind to get GPT out the door. With these LLMs and the secret sauce behind them is becoming less of an academic endeavor and more of a commercial one, perhaps its an inevitable next step.

Re: OpenAI’s policies hinder reproducible research on language models

#63

Post by fullshark »

loveparade wrote: post #58
tourist2d wrote: post #53

Earlier quoted context omitted.

You say: OpenAI really seems to believe that not publishing details will save them from competition. Then say: The only competitive advantage that OpenAI has here is a headstart of 6-12 months It's almost as if they want to keep this advantage, huh? Blows my mind how business illiterate some HN commenters are.

A headstart doesn't matter unless you can keep it. The point is that there are many mort smart people and resources outside of OpenAI and there are inside of OpenAI. If they focus their efforts, they will easily catch up. A headstart is not a competitive moat like network effects are. Go and try to raise money for your startup from a VC and tell them "well, everyone is doing the same as us, but we started 6 months ea…

So OpenAI should just give up and release all their trade secrets? To what purpose?

Re: OpenAI’s policies hinder reproducible research on language models

#64

Post by sva_ »

Eji1700 wrote: post #43

Open AI has been doing sketchyish things long before Chat GPT, and I think it's something people are eventually going to notice more and more (then again people were swearing that Musk walked on water for waaaaaay too long given his actions so fuck if I know). They're 100% marketing FIRST. I don't think they'll outright lie, but they will absolutely screw with their data in such a way to make it look waaay more impre…

My comment might've seemed like I judge them for trying to make a profit - I don't, since there's nothing wrong with that. I was more pointing to the fact that they probably need to make a profit, rather sooner than later, so they aren't shackled by M$ and can be an independent company.

Re: OpenAI’s policies hinder reproducible research on language models

#65

Post by loveparade »

sva_ wrote: post #22

Since OpenAI didn't release the parameter count of GPT-4, I've been wondering/doubting if it is really much bigger than GPT-3. The release of GPT-3.5 has shown that they've found ways of drastically cutting down compute costs (an order of magnitude) while maintaining or even improving the quality of the model's outputs. Perhaps the reason that they didn't release the specifics of GPT-4 might be in part due to them wa…

Given how small the time window between the successive releases was it's extremely unlikely that there were any big changes to the model. Most likely it's just better preprocessed training data, more training data, trained for longer, performance optimizations for attention, or a few changes to layer sizes.

Re: OpenAI’s policies hinder reproducible research on language models

#66

Post by ftxbro »

Historically, researchers at some of the biggest tech companies had permission to publish their results. Presumably it was mutually beneficial; many researchers held dual positions in academia and industry, and publishing cool models could attract good researchers to the company.

But stuff got real. They discovered a path to super-human cognition that scales directly with money and computer chips. Now these companies are closing their public academic work, looking for partnerships with companies like nvidia, and firing large swaths of employees.

Re: OpenAI’s policies hinder reproducible research on language models

#67

Post by nutanc »

atleastoptimal wrote: post #32

I understand any individual's company anti-competitive measures. OpenAI looks at Google the same way Apple looked at IBM in the 80s. What I'm worried about is a lot of the talk about guarding models, public safety and misuse of models will end up leading every big company to pull public access of their APIs. We might look at 2022-2023 as a brief golden age when regular people could use stuff like GPT-4 before it was…

You guard your API, I guard my data.

Re: OpenAI’s policies hinder reproducible research on language models

#68

Post by teruakohatu »

sillysaurusx wrote: post #60
loveparade wrote: post #48

What's most surprising to me is that OpenAI really seems to believe that not publishing details will save them from competition. Everyone knows how these models work, and while I'm sure there is a bunch of "secret sauce" that OpenAI has built for training and fine-tuning, it's ridiculous to believe that the research community and competitors like Google and Facebook can't figure out the same. They just haven't really…

Au contraire, no one knows how large GPT-4 is, which is the single best predictor of performance (for a model trained to convergence). The GPT-4 paper spent much of its time writing about this — they did some small scale experiments with 1/1000th the compute, then picked a loss level they wanted and trained GPT-4 till it got it. Neither the exact loss level nor the number of parameters are revealed by the paper. Unfo…

For all we know they have hit 500B parameters with some clever unpublished optimisation, which would both give them an edge and if revealed would put a damper on the preveiling belief that LLMs can scale and scale (eg. 3x more params for less than 3x performance).

As you say, there is absolutely no way for us to find out.

Re: OpenAI’s policies hinder reproducible research on language models

#70

Post by snickerbockers »

atleastoptimal wrote: post #32

I understand any individual's company anti-competitive measures. OpenAI looks at Google the same way Apple looked at IBM in the 80s. What I'm worried about is a lot of the talk about guarding models, public safety and misuse of models will end up leading every big company to pull public access of their APIs. We might look at 2022-2023 as a brief golden age when regular people could use stuff like GPT-4 before it was…

>a lot of the talk about guarding models, public safety and misuse of models

The stuff about models potentially being misused is just their public justification to look like the good guys. They're not going to withhold their technology because they don't want it to be misused, they're withholding it because they want control over who misuses it. Of course, it won't be called "misuse" when the right parties are doing it.

Post reply on HN