Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 details leaked?

threadreaderapp.com

Post reply on HN
Print view View original

371–380 of 648 posts

Re: GPT-4 details leaked?

#371

Post by DennisP »

DeathArrow wrote: post #258
jona-f wrote: post #167

Earlier quoted context omitted.

Yes, they are doing the improving, but then you need loads of money to do the learning no university can afford. So now big tech is hiring promising university researchers for good money to scale up their research. This could be solved by massive decentralization where millions of users provide compute with their gpus and i think it will be at some point, cause i believe foss is more powerful than this openai bs. The…

> There are people working on this, but afaik the techniques aren't quite there. You need a different kind of model with much more parallelization then what is currently used. What if crypto is switching from mindless hashing as proof-of-work to training AI models as proof-of-work? That would mean suddenly big computing resources are available.

For blockchains using proof-of-work you need two things:

1) The work has to be very hard to do (and quantifiably so), but very easy to verify.

2) The block's transactions have to be an input to the computation, and it has to be impossible to get the same output with a modified set of transactions.

Cryptographic hash functions fulfill both of these requirements. Almost nothing else does.

However, if blockchains switch to proof-of-stake, then the GPUs previously dedicated to that blockchain are available for other purposes. But the biggest GPU-based blockchain already did that, and Bitcoin uses specialized hardware that can't do anything other than Bitcoin's hash function.

Re: GPT-4 details leaked?

#372

Post by criley2 »

mensetmanusman wrote: post #319
YeGoblynQueenne wrote: post #229

Earlier quoted context omitted.

>> We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us. Just to be clear, there's no science being kept secret because there is no science being done. OpenAI's is a feat of engineering, borne aloft by a huge budget supporting a large team whose expertise lies in tuning neural net systems, and not in doing science. Machine learning, as it is practiced to…

{Hypothesis, test, loop} is the scientific method, and I can guarantee it is being used when fine tuning an LLM.

It is not the scientific method at all. You didn't even include an analyze or a publish step, two critical components to the scientific method. The point is that science is a thoughtful, methodical, recorded, repeatable process that is scrutinized by not just your friends but by the entire world, including those who compete against you. In science, it's normal for your competitor to recreate your experiment exactly to see if you did it as well as you claim. Or to push it further.

What OpenAI is doing doesn't even resemble science at all.

Re: GPT-4 details leaked?

#373

Post by londons_explore »

xeckr wrote: post #135

If this is true, then: 1. Training took 21 yottaflops. When was the last time you saw the yotta- prefix for anything? 2. The training cost of GPT-4 is now only 1/3 of what it was about a year ago. It is absolutely staggering how quickly the price of training an LLM is dropping, which is great news for open source. The google memo was right about the lack of a moat.

> The google memo was right about the lack of a moat.

5 months on, and nobody has yet beaten their result quality. I think there is a moat.

Also, I think for many usecases, smarter is better. If a few cents can buy a more accurate answer, then it is always worth paying those few cents. So, while more hardware and more data can train a bigger better model, then that is the moat.

Re: GPT-4 details leaked?

#374

Post by mdp2021 »

TeMPOraL wrote: post #351
mdp2021 wrote: post #323

Earlier quoted context omitted.

...but in a technical context, not in a scientific one. One thing is identifying (age of copper; age of bronze) the best ways of smelting ore to obtain the metal through trial and error, another is to try and understand the nature of materials.

Corporate wants you to find the difference... That is to say, the two things you mention are the same process. "Identifying the best ways of smelting ore to obtain the metal through trial and error" is the easy part, when you get to pick low-hanging fruits in a field. But as the easy options get cleared out, continuing improvements requires increasingly complex, sophisticated methods - that's where the process transi…

The difference, while moving in "interdependent" directions, is in the purpose: obtaining some sufficient information on how things work versus an actual consideration of the nature of things.

It is not really (fully the same process), because you could (in theory) "early stop" when you have achieved technically sufficient competence - the description of the optimal process -, before the jump to the understanding. Something works - that is the technical side; why it works is the scientific side.

Note that the approach has controversial sides: if you took Newton's "hypoteses non fingo", He refused that jump explicitly:

> hypotheses, whether metaphysical or physical, or based on occult qualities, or mechanical, have no place in experimental philosophy

I.e.: Newton proposed an «experimental philosophy» which stopped short of a "true understanding of the reasons behind phenomena", in order to avoid contexts in which solidity (at his time) could not be expected. "Understanding" as "identifying universal laws" was the best that seemed affordable.

Re: GPT-4 details leaked?

#375

Post by immibis »

potatototoo99 wrote: post #210
KronisLV wrote: post #112

Earlier quoted context omitted.

Huh, more power to those folks then, I guess. But I can easily imagine more conventional forms of entertainment, as well. Like a game of D&D that's narrated by the AI, or a text based adventure set in the Mass Effect universe, Lord of the Rings, Warhammer or any other fandom, really. Maybe like those old Choose Your Own Adventure games. I think some companies are also experimenting with characters in video games that…

Do you seriously think D&D and other choose your adventure games will be more popular than porn? Seriously?

Did you try AI Dungeon before OpenAI killed it?

Yes, people used it for porn.

Re: GPT-4 details leaked?

#376

Post by 29athrowaway »

PostOnce wrote: post #19

"Open" AI, a charity to benefit us all by pushing and publishing the frontier of scientific knowledge. Nevermind, fuckers, actually it's just to take your jobs and make a few VCs richer. We'll keep the science a secret and try to pressure the government into making it illegal for you to compete with us. https://github.com/ggerganov/llama.cpp https://github.com/openlm-research/open_llama https://huggingface.co/TheBlok…

If OpenAI is open, the Congo Free State was a free state.

Re: GPT-4 details leaked?

#377

Post by immibis »

namaria wrote: post #267
ben_w wrote: post #236

Earlier quoted context omitted.

The public position (as opposed to the rumour mills) is that they're not working on a 5, and don't intend to at least until they can figure out how to do it safely.

And safely here means non disruptive to established businesses. If they create a tool so powerful that it can replace entire professional classes, that would just mean people can bypass the employers and get their value directly from an API. OpenAI needs to make sure they have a business model where they can charge enterprise fees and provide value to corporations, not directly to people. ChatGPT was merely a publici…

Huh? They'd sell access to the tool to the employers. They'd make a nice rent, employers would make a nice profit by firing all their employees, and customers would still have to go to employers to get whatever service done.

Re: GPT-4 details leaked?

#378

Post by londons_explore »

jetrink wrote: post #310
YeGoblynQueenne wrote: post #239

Earlier quoted context omitted.

>> The training cost of GPT-4 is now only 1/3 of what it was about a year ago. It is absolutely staggering how quickly the price of training an LLM is dropping, which is great news for open source. The google memo was right about the lack of a moat. That really doesn't change anything at all. The more training large models gets cheaper, the more large corporations are able to train larger models than everyone else. S…

At a certain point though, models become good enough for particular tasks. Once that happens for whatever my application is, I don't care if OpenAI has a model that's twice as good on some metric, because it's overkill for my use-case. I'm going to be happy using a smaller, cheaper model from a competitor.

I think we're far from that point though. For the vast majority of use cases, I always wish that the answers could be more accurate.

Sure - they might be 'good enough' to build a business on. But if a competitor builds their business on top of a more accurate model, their product will work better, and they will win the market.

Re: GPT-4 details leaked?

#379

Post by simcop2387 »

Incipient wrote: post #227
ZiiS wrote: post #209

Earlier quoted context omitted.

Most of the ram is for storing the model once it is loaded it is read only so will not harm an SSD.

It only reads from memory,not swap directly. If it needs to read something from swap, it'll write out something from memory to swap, then read the swap into memory. Reading 1gb of swap, will essentially write 1gb to the ssd too. (rough numbers) Correct me if I misunderstand swap?

That's basically right. I'm not sure if Linux or windows will keep track of the pages it read out of swap to know if they're still there and valid, but there's a better way for this that I think at least ggml supports where it stores a copy of the model unpacked and ready on disk as a cache for doing the work rather than relying on the OS virtual memory to handle it. This should be faster than the OS VMM (though probably not by much) but since it'll know which pieces it needs to leave on disk and where they are it should be much safer as far as writes go since it will know enough to not write multiple times like that.

Re: GPT-4 details leaked?

#380

Post by immibis »

KronisLV wrote: post #112
antupis wrote: post #92

Earlier quoted context omitted.

I bet that open models win in the end because porn. There is already very weird and vibrant community creating "waifus" and tinkering with these models.

Huh, more power to those folks then, I guess. But I can easily imagine more conventional forms of entertainment, as well. Like a game of D&D that's narrated by the AI, or a text based adventure set in the Mass Effect universe, Lord of the Rings, Warhammer or any other fandom, really. Maybe like those old Choose Your Own Adventure games. I think some companies are also experimenting with characters in video games that…

You are describing AI Dungeon, a third-party product using GPT-3 that OpenAI killed about two years ago.
Post reply on HN