Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The genie escapes: Stanford copies the ChatGPT AI for less than $600

newatlas.com

Post reply on HN
Print view View original

91–100 of 173 posts

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#91

Post by Karrot_Kream »

pram wrote: post #58
crooked-v wrote: post #23

Earlier quoted context omitted.

For me the easiest comparison between models is to give it an absurd but entirely possible request, like "Write me a fanfic where the Animorphs battle the truck from Duel, but in the style of Mark Twain". So far nothing else I've tried has done even as well as GPT 3.5 yet, let alone GPT 4.

How exactly do you get it to keep going? Every time I try a prompt like this in the playground it spits out a couple paragraphs and then refuses to generate anything further, even with tokens maxed out.

You can just resubmit the prompt and existing history to have the model continue generating new tokens.

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#92

Post by zamnos »

nico wrote: post #73
xwdv wrote: post #2

Given the high prices of OpenAI offerings it seems it’s better to pirate an AI model before resorting to paying for anything.

How much does it cost to privately fine-tune and run Llama? It’s USD 600 for fine-tuning. Maybe USD 4-5k for a computer that can run it. ChatGPT pro is $20/month. 5k would be 250 months (10+ years) of paid access. Not sure pirating it now adds up.

Correct me if I'm wrong, but that's per-user, unless you all just share an account, or you build out a bespoke API integration. Which is to say if you have 25 developers, you'd spend $5k in 1 month.

The reason to pirate it would to be able to fine-tune the model on your private internal source code repository, assuming you already have an existing large body of work you want to train it and offer SelfHostedCoPilot on your bespoke internal-only DSL that ChatGPT and Copilot has no way of having seen and would undoubtedly hallucinate about by a considerable margin.

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#93

Post by Jcowell »

cjohnson318 wrote: post #90

> It seems these godlike AIs are already frighteningly cheap and easy to replicate. "godlike"? Really? I'm not religious, but this seems like an overreaction for something that has no agency.

What if creation was a result of a lucky happen-by-chance hallucination ?

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#94

Post by crazygringo »

cjohnson318 wrote: post #90

> It seems these godlike AIs are already frighteningly cheap and easy to replicate. "godlike"? Really? I'm not religious, but this seems like an overreaction for something that has no agency.

If it's a shorthand for omniscience then I can see how it makes sense. A bit hyperbolic though for sure.

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#96

Post by homarp »

braingenious wrote: post #79
disgruntledphd2 wrote: post #66

Earlier quoted context omitted.

Alpaca wasn't great. The 13b and 30b models are much better, but just for sentence completion. Personally, I think that the RLHF does make a big difference but maybe it's a bug in the quantization code as suggested up thread.

I’m also a bit confused by the quantization thing. Why exactly is everybody running the same program on the same file? Why not just include the quantized weights? It seems like if somebody figured out the “correct” way to quantize the 7b weights it would make way more sense to just torrent the output rather than distribute a fixed program.

quantization takes lots of RAM:

https://github.com/qwopqwop200/GPTQ-for-LLaMa/blob/main/READ... says llama-13B takes 42GB and 33B takes more than 64GB...

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#97

Post by UncleOxidant »

superkuh wrote: post #8

Hardly. I've played a lot with the 7,13, and 30B llamas as well as the 7 and 13B alpacas fine tuned by Stanford. They do not have emergent abilities like being able to generate rhymes or, say, represent a movie plot as emoji. Even openai's old text-davinci-003 (gpt3.5, but text completion, not the chat ones) far outperforms them. That said, I have hopes for a 65B 3-bit quantized alpaca-fine tuned. We'll see when some…

> the alpaca 7B _4-bit_ [and presumably also 4bit for the 13B, 30B and larger parameter sets]

This is the wild card here, though, isn't it? OpenAI's chatGPT likely uses more than 4 bits for it's parameters. IIRC the original LLaMA params were 16bit floats and they were quantitized down to 4bit - considering that large amount of compression, they sill do pretty OK, but not as good as chatGPT. I wonder how the alpaca/LLaMA models would do with 16bit floating point params (as they were originally trained)? What if they would have gone with 8 bits for the params as a compromise?

EDIT: Come to think of it, unless you're using vectorized ops on a CPU, 4 bit and 8 bit math is going to run at the same speed (for most popular CPUs), is it not? So why did they go all the way down to 4 bits instead of stopping at 8 bits (other than to make the param files 1/2 the size)?

EDIT2: looking through the alpacca.cpp code and there is mention of AVX, AVX2, AVX512 (and NEON on ARM) so it probably is taking advantage of vectorized ops where that's possible.

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#98

Post by simonw »

Related, my post "Could you train a ChatGPT-beating model for $85,000 and run it in a browser?" https://simonwillison.net/2023/Mar/17/beat-chatgpt-in-a-brow...

I think you can train LLaMA 7B (the model underlying Alpaca) for around $82,000, based on the Meta Research paper about it. Then you can fine-tune it ala Alpaca for a few hundred dollars more.

My wilder speculation is that, if you can shrink the model down to 4GB with llama.cpp 4bit quantization, it may be possible to run it entirely in the browser (ala Stable Diffusion from the other day).

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#99

Post by kmeisthax »

sfriedr wrote: post #78
zamnos wrote: post #42

Earlier quoted context omitted.

Maybe. Certainly in the past, before the world was aware LLMs on the level of ChatGPT were possible with today's technology. OpenAI's chosen not to release any real details about GPT-4, so we don't actually know what it would take to train a model of equivalent quality, especially considering training isn't a one-shot. Multiple training runs easily add up training costs. So training for a 12-figure parameter size mod…

> OpenAI's chosen not to release any real details about GPT-4 Actually, they have release some details about it, in this 99-page technical report https://arxiv.org/abs/2303.08774 (which is actually two papers stitches together, once you read it; oddly enough using different fonts). But I'm not sure if this content qualifies as "real details".

The intro to that paper specifically says:

> Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar. We are committed to independent auditing of our technologies, and shared some initial steps and ideas in this area in the system card accompanying this release. We plan to make further technical details available to additional third parties who can advise us on how to weigh the competitive and safety considerations above against the scientific value of further transparency.

In other words, "Stable Diffusion wasn't supposed to happen, so we're making all our methodology trade secret[0], if you want to Do Science then agree to this massive NDA and have enough skin in the game for us to cut you."

[0] Presumably at some point OpenAI will have to 'relent' to independent discovery by patenting AI architectures and refusing to license them

Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600

#100

Post by TedDoesntTalk »

crooked-v wrote: post #85
ianlevesque wrote: post #77

Earlier quoted context omitted.

That's ok, just ask ChatGPT to explain them to you.

ChatGPT 4 input: > Can you give separate summaries of the Animorphs book series and the film Duel? The reply: > Animorphs Book Series Summary: The Animorphs series, written by K.A. Applegate, is a popular young adult science fiction book series spanning 54 books and several companion novels. The story revolves around five teenagers—Jake, Rachel, Tobias, Cassie, and Marco—who are granted the power to morph into any an…

Duel is a great film but requires patience compared to a lot of today's films. There's not much action for quite some time -- just a man driving through the desert. Even when things pick up, it's still slow pacing by today's standards. It's from another era, and I doubt anyone would make a film today with the same pacing.
Post reply on HN