Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 details leaked?

threadreaderapp.com

Post reply on HN
Print view View original

451–460 of 648 posts

Re: GPT-4 details leaked?

#451

Post by gitfan86 »

jncfhnb wrote: post #428
YeGoblynQueenne wrote: post #229

Earlier quoted context omitted.

>> We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us. Just to be clear, there's no science being kept secret because there is no science being done. OpenAI's is a feat of engineering, borne aloft by a huge budget supporting a large team whose expertise lies in tuning neural net systems, and not in doing science. Machine learning, as it is practiced to…

Sure. But remember also that the standard “science” is largely a broken model. Academia sucks. It’s honestly bad. It’s overflowing with papers that are misleading, irrelevant, or fraudulent due to a mix of poor stats knowledge, bad incentives, and a failure to organize effectively. While people praise the scientific method, the majority of achievements we attribute to “science” are not derived from guess and check gr…

Right. In historical context the scientific method was a huge improvement over the previous method of understanding the world, which was mostly religion.

But today, academia is a victim of organizational and political capture, making it less competitive for talent.

Re: GPT-4 details leaked?

#452

Post by RC_ITR »

why_only_15 wrote: post #205
RC_ITR wrote: post #165

Earlier quoted context omitted.

The thing about these models is compute scales quadratically with model dimensionality and memory scales quadratically with sequence length. We are nowhere near diminishing returns for either variable, so sure current models maybe scale quickly but the cutting edge will want as much compute as possible for a long time. That’s kind of the humor of everyone saying this leak somehow leaves OpenAI vulnerable. The work is…

Memory does not scale quadratically with sequence length.

During training, you have to store a dot product of Q and V that has dimension Ncrt^2.

That's quadratic scaling, no?

Re: GPT-4 details leaked?

#453

Post by YetAnotherNick »

astrange wrote: post #172
YetAnotherNick wrote: post #32

Earlier quoted context omitted.

Even if I just go with Sam Altman's public comment, I would have came to similar conclusion: GPT-4 is big and it is hard to make it is faster. The secret sauce and moat lies in data though. I have heard rumour that they have paid competitive coders to write and annotate code with information like complexity for them.

GPT4 can diagram sentences using link grammar parsing ( https://www.link.cs.cmu.edu/link/ ) which is obscure enough I really don't think they've generated data for it. So it can get pretty good without that.

It's obvious they use data from github and other places. I am talking about extra 0.00..1% very high quality data they (likely)created.

Re: GPT-4 details leaked?

#454

Post by s3p »

TradingPlaces wrote: post #336
PostOnce wrote: post #19

"Open" AI, a charity to benefit us all by pushing and publishing the frontier of scientific knowledge. Nevermind, fuckers, actually it's just to take your jobs and make a few VCs richer. We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us. https://github.com/ggerganov/llama.cpp https://github.com/openlm-research/open_llama https://huggingface.co/TheBlok…

Keep in mind, that was the idea originally. Then in 2018 Elon decided he wanted to be CEO, the board rejected that idea for now obvious reasons, and he reneged on 90% of a promised $1b donation. The only way forward was to become a for-profit company and do a normal funding round, which is what happened with Microsoft. Elon’s rug pull is why this happened.

And to this day Elon slanders OpenAI for taking profit.

It’s strange because their corporate structure is highly unique. They generate fixed returns on investments, so investors can’t make that much money. Also, the profit-generating division is wholly owned and controlled by the nonprofit. I don’t understand why Elon and others have so many problems with this.

Re: GPT-4 details leaked?

#455

Post by gitfan86 »

nix0n wrote: post #440
mensetmanusman wrote: post #319

Earlier quoted context omitted.

{Hypothesis, test, loop} is the scientific method, and I can guarantee it is being used when fine tuning an LLM.

The reason that this is Engineering as opposed to Science, is that the hypothesis is just, "hey, maybe this will work". Nobody has really explained why it works.

Which is a fundamental flaw in science.

Most of AI academics have spent their career theorizing complex algorithms or complex explanations of intelligence.

But the engineers have built large enough Neural Networks to give us data points that show intelligence is emergent out of relatively simple components.

Unsurprisingly, the people who believed they were the smartest were the least likely to explore the possibility that human intelligence isn't general, but specialized.

Echos of the academics building heliocentric models of the universe centuries ago.

Re: GPT-4 details leaked?

#456

Post by JKCalhoun »

machina_ex_deus wrote: post #432
YeGoblynQueenne wrote: post #229

Earlier quoted context omitted.

>> We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us. Just to be clear, there's no science being kept secret because there is no science being done. OpenAI's is a feat of engineering, borne aloft by a huge budget supporting a large team whose expertise lies in tuning neural net systems, and not in doing science. Machine learning, as it is practiced to…

I'm tired of science as a religion. People treat it as some gospel, like if you check out some criterions you're suddenly "scientific" and instantly get a sense of validity and authority that you shouldn't logically get. I judge things as "what you can do", not "what can you predict". The only demonstration of knowledge and understanding is being able to do something. Not predict. Not "scientific method" and ridiculo…

Science as a rear-view mirror.

Re: GPT-4 details leaked?

#457

Post by s3p »

blitz_skull wrote: post #392
0xDEF wrote: post #328

Earlier quoted context omitted.

I think most of HN has only tried ChatGPT with GPT-3.5 in December 2022 when the OpenAI servers were getting hammered. That is why they're impressed by these slow and low quality local models and mistakenly think they're on par with OpenAI's offerings. Honestly even reddit and teenagers on TikTok have a more accurate view on OpenAI vs. local LLMs than HN.

There’s not a single thing out there that even comes close to GPT-4. Not one that I’ve used anyway. Benchmarks be damned, it’s the experience that matters and I’ve yet to have an LLM blow my mind the way GPT-4 does.

Have you used Claude? I regularly use them instead of GPT4 because of the larger context window. 4 is still useful when I want to give instructions to the system (Claude will refuse to do a lot), but generally the responses seem on par with each other.

Re: GPT-4 details leaked?

#458

Post by lorepieri »

machina_ex_deus wrote: post #432
YeGoblynQueenne wrote: post #229

Earlier quoted context omitted.

>> We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us. Just to be clear, there's no science being kept secret because there is no science being done. OpenAI's is a feat of engineering, borne aloft by a huge budget supporting a large team whose expertise lies in tuning neural net systems, and not in doing science. Machine learning, as it is practiced to…

I'm tired of science as a religion. People treat it as some gospel, like if you check out some criterions you're suddenly "scientific" and instantly get a sense of validity and authority that you shouldn't logically get. I judge things as "what you can do", not "what can you predict". The only demonstration of knowledge and understanding is being able to do something. Not predict. Not "scientific method" and ridiculo…

This is the reason why I left academia for startups... Seems like a better way (not perfect by any means) to innovate, actually doing things.

PS: I would be happy to connect, you can find my socials in my bio.

Re: GPT-4 details leaked?

#459

Post by hyperthesis »

criley2 wrote: post #372
mensetmanusman wrote: post #319

Earlier quoted context omitted.

{Hypothesis, test, loop} is the scientific method, and I can guarantee it is being used when fine tuning an LLM.

It is not the scientific method at all. You didn't even include an analyze or a publish step, two critical components to the scientific method. The point is that science is a thoughtful, methodical, recorded, repeatable process that is scrutinized by not just your friends but by the entire world, including those who compete against you. In science, it's normal for your competitor to recreate your experiment exactly t…

The publish step of the scientific method always seemed out of place to me - it means you can't discover anything on your own, which doesn't seem correct to me.

However, to discover something new (not just new to you) does require contact with established knowledge. And if you don't publish, you can't be very sure that it really is "new"; and it does not become part of established knowledge - it has not been discovered.

i.e. the concept of discovering new knowledge means it cannot be done on a desert island.

Re: GPT-4 details leaked?

#460

Post by blueprint »

dylan604 wrote: post #366
blueprint wrote: post #335

Earlier quoted context omitted.

I hate to break it to you but that is part of science. Perhaps the major part of it too. Hypothesis: I can build a cabinet with these materials which will bear some load range. Experiment: I built it and it obviously works. Now change cabinet to particle accelerator that a giant team of other theorists and engineers designed. Am I not doing science by participating in building it? So experimental scientists arent sci…

To me, you're describing the differences between a cook and a chef. Just because you've built something with well known methods doesn't make you a scientist. You didn't come up with anything new. In fact, we're starting to sound a lot like Apple. Apple is (in)famous for taking ideas that someone else did all of the hard work of developing and proving to work, and then take various ideas like that to combine into an a…

semantics, I say. cooks are "chiefs" of certain things otherwise they'd have no value on a team. unless you're saying that chefs setting the menu means theyre more of a scientist. at worst you're saying e.g. scientists are like chefs in that their used their creativity to make up the standard model. at best you're saying chefs are theorists and cooks are experimental scientists which is , again, to my own point.
Post reply on HN