I bet you could “exfiltrate” an LLM relatively cheaply by using LLM A to generate training data for LLM B.
The genie escapes: Stanford copies the ChatGPT AI for less than $600
51–60 of 173 posts
Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600
#52This is why it's not possible to slow down or "stop" AI: once the problems are solved the solutions turn out to be trivial to replicate. All it takes is compute.
you say all it takes compute like that is trivial - chatGPT would have a hard time without Microsoft's support via Azure
There are lots of places to get compute, including Chinese cloud providers...
The genie really is out of the bottle now.
This is a lot like pharmaceuticals. The initial investment in a new medication is enormous. The price of each pill is trivial, to the extent that every drugstore chain is able to supply a generic in-house brand.
Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600
#53Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600
#54Earlier quoted context omitted.
For me the easiest comparison between models is to give it an absurd but entirely possible request, like "Write me a fanfic where the Animorphs battle the truck from Duel, but in the style of Mark Twain". So far nothing else I've tried has done even as well as GPT 3.5 yet, let alone GPT 4.
> Write me a fanfic where the Animorphs battle the truck from Duel, but in the style of Mark Twain Whoa. I want to read this! Duel - what a great film. Twain - amazing writer. Animorphs - published after my teen years but sounds like a great story!
Highly recommend Animorphs. As an adult the books can be read through pretty quickly, but the story really holds up. Deals with some heavy topics of genocide, enslavement (even exploring the justifications of the enslavers), PTSD among other topics but was still considered a kids book.
It becomes obvious in the middle when some of the books were written by ghost writers, but the books are so easy to read I don't really recommend skipping them. If you must you could probably get away with reading the first ten, last ten, but should definitely read all of the Chronicle books.
Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600
#55Earlier quoted context omitted.
While that's true, it's basically inevitable now that at some point personal hardware will be powerful enough for enthusiasts to run home bots comparable to GPT-3, and even that by itself would drastically change a lot things.
Running isn’t necessarily the issue. The moat is creating a high-quality model like OpenAI has, which (and here the article is mistaken) doesn’t seem to be easily reproducible.
That's going to get easier too. Stanford can already get this far for $600, so soon after the major GPT-based chat AIs were released. Imagine how much better it will get with just a little bit more time.
Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600
#56Earlier quoted context omitted.
I'm not convinced that the GPT-4 API actually works? It's been giving me very different answers than the chat interface. For example, the chat interface says it's GPT-4 if you ask it, but the API says it's GPT-3 (and bills as GPT-4).
Why would the LLM model know what version it actually is? It was trained on data from before that model version even existed.
- What version model are you?
- I am based on the GPT-4 architecture, which is an iteration of OpenAI's Generative Pre-trained Transformer models. My training data includes information up to September 2021.
- How do you know your version?
- As an AI language model, I am programmed with information about my own architecture and the version I am based on. My creators at OpenAI provided me with this knowledge, which helps me to give context to users about my capabilities and limitations. In this case, I am based on the GPT-4 architecture, an iteration of OpenAI's Generative Pre-trained Transformer models.
Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600
#57Earlier quoted context omitted.
I'm not convinced that the GPT-4 API actually works? It's been giving me very different answers than the chat interface. For example, the chat interface says it's GPT-4 if you ask it, but the API says it's GPT-3 (and bills as GPT-4).
If you're satisfied with the results who cares what it thinks it is
I'm not. It's worse than the chat version.
Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600
#58Hardly. I've played a lot with the 7,13, and 30B llamas as well as the 7 and 13B alpacas fine tuned by Stanford. They do not have emergent abilities like being able to generate rhymes or, say, represent a movie plot as emoji. Even openai's old text-davinci-003 (gpt3.5, but text completion, not the chat ones) far outperforms them. That said, I have hopes for a 65B 3-bit quantized alpaca-fine tuned. We'll see when some…
For me the easiest comparison between models is to give it an absurd but entirely possible request, like "Write me a fanfic where the Animorphs battle the truck from Duel, but in the style of Mark Twain". So far nothing else I've tried has done even as well as GPT 3.5 yet, let alone GPT 4.
How exactly do you get it to keep going? Every time I try a prompt like this in the playground it spits out a couple paragraphs and then refuses to generate anything further, even with tokens maxed out.
Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600
#59Earlier quoted context omitted.
you say all it takes compute like that is trivial - chatGPT would have a hard time without Microsoft's support via Azure
While that's true, it's basically inevitable now that at some point personal hardware will be powerful enough for enthusiasts to run home bots comparable to GPT-3, and even that by itself would drastically change a lot things.
Governments can ban powerful devices, as they can ban guns, bombs, and such.
Re: The genie escapes: Stanford copies the ChatGPT AI for less than $600
#60Earlier quoted context omitted.
3 bits? Is that for all weights in the network?
As far as I know, yes. https://arxiv.org/abs/2210.17323 "Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline." This would be 175 billion 3 bit weights instead of 175 billion 16 (or 32!) bit weights. It massively reduces the size of the mod…
The number of parameters stays the same, but the amount of information encodable by those parameters is not the same.