Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 details leaked?

threadreaderapp.com

Post reply on HN
Print view View original

181–190 of 648 posts

Re: GPT-4 details leaked?

#181

Post by numpad0 »

TeMPOraL wrote: post #125
BelleOfTheBall wrote: post #119

Earlier quoted context omitted.

That is a very dangerous way to approach copyright laws. They are definitely abused by corporations like Disney, infamously so, but abolishing them is absolutely not the answer. Art makers are already struggling en masse, taking away their ability to earn money off their work isn't an answer, especially if it's just to train a predictive text generator.

Counterpoint: there's way too much "art makers", copyright is keeping the quality ceiling down, good content loses monetization game with spam - all while that "predictive text generator" is, for better or worse, pretty much the most magnificent piece of technology invented this century, and - for better or worse - it's likely to become next major shift to economy and life.

- HN on social media : The powers are too centralized, future is decentralization, question is how

- HN on free software: is good

- HN on copyright : WAY TOO MUCH PEASANTS CLAIMING INDIVIDUAL RIGHTS, RIGHTS THAT ARENT EVEN REAL, ART BE CENTRALIZED FOR MAXIMUM MONOPOLY

Re: GPT-4 details leaked?

#182

Post by Aachen »

krackers wrote: post #17
CSMastermind wrote: post #2

Previously posted about here: https://news.ycombinator.com/item?id=36671588 and here: https://news.ycombinator.com/item?id=36674905 With the original source being: https://www.semianalysis.com/p/gpt-4-architecture-infrastruc... The twitter guy seems to just be paraphrasing the actual blog post? That's presumably why the tweets are now deleted. --- The fact that they're using MoE was news to me and very interesting. I…

Interesting on a meta point that the more clickbaity title "GPT-4 details leaked" won out over the more dispassionate but drier "GPT-4 Architecture, Infrastructure, Training Dataset, Costs".

When choosing titles for my own submissions, yeah, the accurate title that HN says they desire gets no votes whatsoever. Any clickbait on here, people bring upon themselves (and this isn't even a clickbait-level title)

Re: GPT-4 details leaked?

#183

Post by johnnyworker »

koheripbal wrote: post #137
juliensalinas wrote: post #100

Earlier quoted context omitted.

LLaMA 30B or 60B can be very impressive when correctly prompted. Deploying the 60B version is a challenge though and you might need to apply 4-bit quantization with something like https://github.com/PanQiWei/AutoGPTQ or https://github.com/qwopqwop200/GPTQ-for-LLaMa . Then you can improve the inference speed by using https://github.com/turboderp/exllama . If you prefer to use an "instruct" model à la ChatGPT (i.e. tha…

4-bit quantization removes a lot of the model's sophistication, and 60B parameters is still smaller than what GPT4 is using.

The point is that it's infinitely better in not being there "just to take your jobs and make a few VCs richer". Nobody even claimed it's more performant. It's like the difference getting nothing, but keeping your land, and getting glass pearls, losing your land. You have to completely ignore the meat of the argument to even pretend there is a contest.

And this is without considering what happened if we stopped feeding hostile actors and supported ourselves, instead of keeping to do the reverse. Not just here and there, but consistently for decades.

Re: GPT-4 details leaked?

#184

Post by K0balt »

downWidOutaFite wrote: post #85
PostOnce wrote: post #38

Earlier quoted context omitted.

It's several reasons, first it's the lies and the abuse of a charity, that wouldn't be an issue if they had began as a private company instead of robbing a charity. But secondly, even if they were a private company, it's dishonest and reprehensible to claim to congress that you want to "protect the public" when you really only give a shit about protecting your moat, I'm not happy about that either. I'm also tired of…

The "regulatory capture" conspiracy theory makes no sense to me. It takes 9 figures in cash to create and run one of these super big models. Only big tech was ever going to create them, and big tech is already very experienced at navigating regulation, regulation wasn't ever going to stop them from competing. And in general, our democracy works better than the nihilist libertarians give it credit for.

I interpreted openAIs regulatory capture bid as more of an attempt to create competition-hostile regulation than an attempt to reduce regulation to cut costs.

Re: GPT-4 details leaked?

#185

Post by maxwin »

wiz21c wrote: post #148
kyledrake wrote: post #65

Earlier quoted context omitted.

Unfortunately I've found the current OSS models to be vastly inferior to the OpenAI models. Would love to see someone actually get close to what they can do with GPT-3.5/4, except capable of running on commodity GPUs. What's the most impressive open model so far?

What are all the researchers in universities doing ? Couldn't they improve these models (they do have big brains after all) with tax payer's money and put the results under some cool open source license...

they produce papers that contain (sometimes) useful ideas. They don't produce code (other than proof of concept) and they certainly can't afford to train a large model

Re: GPT-4 details leaked?

#186

Post by fragmede »

JimmyRuska wrote: post #178
theLiminator wrote: post #170

Earlier quoted context omitted.

The real moat is an abundance of high quality data.

Well open AI raised eye brows by crawling the internet and using everyone's data to make a commercial product One day some new startup will train on all of libgen and torrent networks, but it will be very hard to prove. You'll keep getting these gaps up in questionable morality and legality, and even openai will complain about playing fair

Google Classroom, teenager's essays, written by humans, for learning what it means to be human, and graded by humans, is a richer dataset than anything else I can think of that anyone else couldn't get their hands on.

Re: GPT-4 details leaked?

#187

Post by dacryn »

ed_mercer wrote: post #142
Vecr wrote: post #123

Earlier quoted context omitted.

You can run that model (Wizard-30) on a computer with 64 gigabytes of RAM (or smaller, I don't know how tight you can cut it). You obviously want fast RAM and a good CPU, but you don't need a GPU.

AFAIK you can get away with a swapfile, no need for large amounts of RAM.

wont that nearly kill your ssd if you do it for extended periods of time?

Re: GPT-4 details leaked?

#188

Post by baq »

theLiminator wrote: post #170
xeckr wrote: post #135

If this is true, then: 1. Training took 21 yottaflops. When was the last time you saw the yotta- prefix for anything? 2. The training cost of GPT-4 is now only 1/3 of what it was about a year ago. It is absolutely staggering how quickly the price of training an LLM is dropping, which is great news for open source. The google memo was right about the lack of a moat.

The real moat is an abundance of high quality data.

Yeah they have the internet from before LLMs were used for anything, so the data is not poisoned. Not unlike carbon dating becoming useless for estimating age of anything made after nuclear atmospheric tests, or low-background steel.

Re: GPT-4 details leaked?

#189

Post by hoseja »

antupis wrote: post #92
KronisLV wrote: post #76

Earlier quoted context omitted.

> You can use the above without paying OpenAI. You don't even need a GPU. There are no license issues like with the facebook llama. I actually wrote about getting an LLM chatbot up and running a while ago: https://blog.kronis.dev/tutorials/self-hosting-an-ai-llm-cha... It's good that the technology and models are both available for free, and you don't even need a GPU for it. However, there are still large memory requ…

I bet that open models win in the end because porn. There is already very weird and vibrant community creating "waifus" and tinkering with these models.

The (even pre-emptive!) opposition and censorship seems way stronger this time around than with previous technologies. Like instead of ignoring the porn (or even profiting from it), they are trying very hard to make it impossible.

Re: GPT-4 details leaked?

#190

Post by downWidOutaFite »

K0balt wrote: post #184
downWidOutaFite wrote: post #85

Earlier quoted context omitted.

The "regulatory capture" conspiracy theory makes no sense to me. It takes 9 figures in cash to create and run one of these super big models. Only big tech was ever going to create them, and big tech is already very experienced at navigating regulation, regulation wasn't ever going to stop them from competing. And in general, our democracy works better than the nihilist libertarians give it credit for.

I interpreted openAIs regulatory capture bid as more of an attempt to create competition-hostile regulation than an attempt to reduce regulation to cut costs.

My point is that openai's competitors have no problem handling regulations.
Post reply on HN