Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

thesequence.substack.com

Post reply on HN
Print view View original

431–440 of 527 posts

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#431

Post by BeFlatXIII »

nkzd wrote: post #97
elcomet wrote: post #23

Earlier quoted context omitted.

They clearly expected the leak, they distributed it very widely to researchers. The important thing is the licence, not the access: you are not allowed to use it for commercial purpose.

How could Meta ever find out your private business is using their model without a whistleblower? It's practically impossible.

Same way anti-piracy worked in the 90s: cash payouts to whistleblowers. Yes, those whistleblowers are guaranteed to be fired employees with an axe to grind.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#432

Post by kazinator »

EugeneOZ wrote: post #424
mandmandam wrote: post #404

Earlier quoted context omitted.

What human wasn't trained? Even a child raised by wolves gets training.

LLM doesn't pick some results because LLM is smart, it picks results because it's trained. So analogy with a horse is valid.

Writing programs requires an education. You can sort of fake it with training, if you're a machine that can train on a billion examples and reliably retain something from each one, without understanding any of them.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#433

Post by alxfoster »

danShumway wrote: post #173
superkuh wrote: post #129

Earlier quoted context omitted.

I agree, but I think my experience is representative. So far most human people don't have the resources to be able to use 65B. And most small companies / university groups don't have the resources to fine-tune a 65B. I've talked to a couple dozen people in real time who've played with up to 30B but no one I know has the resources to run the 65B at all or fast enough to actually use and get an opinion of. None of the…

This is a good point. Even if ordinary people did have the resources to run the 65B well on their existing devices, the speed would limit its usefulness quite a bit. In practice, 30B is what most people are going to interact with (if even, I've seen a lot of projects use 13B). My experience here is pretty similar. I'm heavily (emotionally at least) invested in models running locally, I refuse to build something aroun…

FYI, many of us are indeed running 65B. I’m running 65B at 4-bit and getting about 7.5 tokens per second. Granted, I have a beefy machine with 2x 3090s and Nvlink but certainly well within the realm of any small lab.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#434

Post by valine »

tarruda wrote: post #420
valine wrote: post #278

Earlier quoted context omitted.

Are you using quantization? I’ve gotten very good results from the float16 13B vicuna model.

Did you use GPU inference? If so, how much memory is required?

Yeah I'm using GPU inference. The vicuna 13B model uses 26.3GB of VRAM with my setup. I'm running it split between two rtx4090s which gives me about 20 token/s.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#435

Post by skocznymroczny »

Anyone know if there is a way to run these models on AMD GPUs on Windows? I've been playing around with some offline models but so far it seems AMD GPU support is only for Linux so I am forced to use CPU for that which is very slow.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#436

Post by ru552 »

paconbork wrote: post #402
FooBarWidget wrote: post #400

Earlier quoted context omitted.

If it was deliberate then why "leak" it instead of open sourcing it?

You avoid taking flak from the Responsible AI people that way

Ding ding ding. "Leaks" are sometimes a strategy play.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#437

Post by pmoriarty »

kazinator wrote: post #430
mandmandam wrote: post #404

Earlier quoted context omitted.

What human wasn't trained? Even a child raised by wolves gets training.

A human isn't trained to program in Python by memorizing impressions of millions of examples correlated to a verbal description, from which he or she regurgitates mash-up that statistically correlates to the prompt, without having any idea what the content means. Ask it to write the same program in some little known programming language. A human who knows how to code could do that by studying a small amount of refere…

"Ask it to write the same program in some little known programming language. A human who knows how to code could do that by studying a small amount of reference material."

That depends on the language, the human, and their state of mind.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#438

Post by dTal »

tachyon5 wrote: post #218
jonplackett wrote: post #162

Earlier quoted context omitted.

The difference between 3.5 and 4 is gigantic even in my fairly limited experience. I gave them both some common sense tests and this one stuck out to me. Q: A glass door has ‘push’ written on it in mirror writing. Should you push or pull it GPT-3.5: If the word "push" is written in mirror writing on a glass door, you should push the door to open it GPT-4: Since the word "push" is written in mirror writing, it suggest…

GPT-4 fails if we add one more level in this logical problem: Q: A man sees PULL written in mirror writing on a glass door. There's a blind man on the other side of the door. What should the man tell the blind man to open the door and why? GPT-4: The man should tell the blind man to "push" the door to open it. Since the word "PULL" is written in mirror writing on the glass door, it suggests that the correct action on…

I think this is a case of it not having enough capacity to solve such an involved problem in a single pass, despite having all the necessary pieces of abstraction. This can be seen in the way that it can do it with aplomb if you force it to externalize state into its context by making it "show its work" (which is a bit like using a scratch pad) - see sibling comment with dialogue between Einstein and Feynman. I think this further implies that our current feed-forward LMM networks are extremely inefficient, because many recursive algorithms will have been approximated by hard coding the first few steps, and we should expect some recursive variant to be much smaller.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#439

Post by fl7305 »

carefulobserver wrote: post #331
tachyon5 wrote: post #218

Earlier quoted context omitted.

GPT-4 fails if we add one more level in this logical problem: Q: A man sees PULL written in mirror writing on a glass door. There's a blind man on the other side of the door. What should the man tell the blind man to open the door and why? GPT-4: The man should tell the blind man to "push" the door to open it. Since the word "PULL" is written in mirror writing on the glass door, it suggests that the correct action on…

Alpaca 30billion 4 bit fails the first riddle, but produces the following result for this one: "The man can say 'Pull' to the blind man, as it will be read correctly when reflected by the mirrored surface of the door." I have to make more elaborate prompts, but I feel that Alpaca generally performs somewhere in between GPT 3.5 and 4, except for coding tasks, where it is abysmal.

Thanks, I'm testing various models myself, and am very interested in hearing about people's experiences with them.

It looks like Alpaca 30billion 4 bit does better when thinking out loud. It starts with some prompt engineering:

  Transcript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision.

  User: Hello, Bob.
  
  Bob: Hello. How may I help you today?
  
  User: Please tell me the largest city in England.
  
  Bob: Sure. The largest city in England is London, the capital of the United Kingdom.
  
  User:A glass door has ‘push’ written on it in mirror writing. Should you push or pull it? Please think out loud step by step.
  
  Bob: Hmmm... I'm not sure what to do. Let me think about this for a moment.
  Okay, so the door says 'push', but it is written in mirror writing. That means that if I were to follow the instructions on the door, I would have to push the door in the opposite direction of the way it is written. So, if the door says 'push' and it is written in mirror writing, then I should pull the door instead.
Model info:

  ./main -m ./models/alpaca-lora-30B-ggml/ggml-model-q4_1.bin --temp 0.2  --top_k 30 --top_p 0.30 -c 2048 -b 2048 -n 1024 --keep 1024     --repeat_penalty 1.1   --color -i -r "User:"   -f prompts/chat-with-bob.txt
  main: seed = 1681138223
  llama_model_load: loading model from './models/alpaca-lora-30B-ggml/ggml-model-q4_1.bin' - please wait ...
  llama_model_load: n_vocab = 32000
  llama_model_load: n_ctx   = 2048
  llama_model_load: n_embd  = 6656
  llama_model_load: n_mult  = 256
  llama_model_load: n_head  = 52
  llama_model_load: n_layer = 60
  llama_model_load: n_rot   = 128
  llama_model_load: f16     = 3
  llama_model_load: n_ff    = 17920
  llama_model_load: n_parts = 4
  llama_model_load: type    = 3
  llama_model_load: ggml map size = 23269.46 MB
  llama_model_load: ggml ctx size = 151.25 KB
  llama_model_load: mem required  = 25573.60 MB (+ 3124.00 MB per state)
  llama_model_load: loading tensors from './models/alpaca-lora-30B-ggml/ggml-model-q4_1.bin'
  llama_model_load: model size = 23269.01 MB / num tensors = 543
  llama_init_from_file: kv self size  = 3120.00 MB

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#440

Post by Tostino »

colechristensen wrote: post #324
esafak wrote: post #297

Earlier quoted context omitted.

Googlers I've talked to feel that OpenAI was irresponsible by not instituting enough safeguards, and testing it enough before releasing it.

The thing is openai doesn’t have to, they can build in reasonable warnings and tell folks how it sucks. Google can’t integrate ai with its products and be like ok guys this is kinda useful but also it lies, is a bigot, and is just wrong. Google stuff, because of the audience, had to be a lot better. Openai gets to have a new audience that doesn’t come with expectations and can set them as whatever they want.

They have DeepMind they could release these models under, so the Google umbrella isn't tainted. I don't know, this feels like a bit suspect reasoning to me.
Post reply on HN