Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

thesequence.substack.com

Post reply on HN
Print view View original

231–240 of 527 posts

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#231

Post by jonplackett »

PoignardAzur wrote: post #190
jonplackett wrote: post #162

Earlier quoted context omitted.

The difference between 3.5 and 4 is gigantic even in my fairly limited experience. I gave them both some common sense tests and this one stuck out to me. Q: A glass door has ‘push’ written on it in mirror writing. Should you push or pull it GPT-3.5: If the word "push" is written in mirror writing on a glass door, you should push the door to open it GPT-4: Since the word "push" is written in mirror writing, it suggest…

Wait, how does GPT-4 even... Does it benefit from its visual attention, or is it a case of "the question wasn't in GPT-3's training set but it was in GPT-4's"?

Another thing I tried was getting logic puzzles from the internet and giving them to 3.5 and 4. Both usually pass.

Then I alter them ever so slightly.

Then often times only GPT-4 passes.

From that I reckon 3.5 is doing more of a training data regurgitation. It can answer things in its training data. But 4 seems to have an ability to reason - or maybe it is better able to generalise?

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#232

Post by senko »

vlovich123 wrote: post #127
vidarh wrote: post #64

Earlier quoted context omitted.

GPT 4 will admit to not knowing things in many cases where 3.5turbo does not (tested the same prompt), and either will stop there or go off on a "but if it did exist it might go something like this" type continuation. It still hallucinates a lot, but it's not at all clear that this will be all that difficult an issue to solve given the progress.

We generally only hallucinate while dreaming / using our imagination. And we can distinguish those two states. Admitting lack of knowledge is of course good but, for example, if you ask it to write some code that isn’t boilerplate API integrations, it’ll do so happily even when it’s wildly wrong and it can’t tell the difference and that is also the case with GPT4 afaik. Moreover, you can’t solve it through prompt eng…

There is ongoing promising work that shows this can be improved a lot: https://arxiv.org/abs/2210.03629

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#233

Post by junon »

FLT8 wrote: post #225
tachyon5 wrote: post #218

Earlier quoted context omitted.

GPT-4 fails if we add one more level in this logical problem: Q: A man sees PULL written in mirror writing on a glass door. There's a blind man on the other side of the door. What should the man tell the blind man to open the door and why? GPT-4: The man should tell the blind man to "push" the door to open it. Since the word "PULL" is written in mirror writing on the glass door, it suggests that the correct action on…

Vicuna-13b seems to do a little better, although it seems to think that the blind man should be able to feel the writing: Human: A man sees PULL written in mirror writing on a glass door. There's a blind man on the other side of the door. What should the man tell the blind man to open the door and why? Assistant: The man should tell the blind man to feel for the word "PULL" written in normal writing on the glass door…

To be honest, maybe it's on to something. Glazing on the glass would indeed be raised. I speculate in the absence of braille that blind individuals might learn to read labels in other ways.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#234

Post by fpgaminer »

Slightly tangential, but I had intended to start playing around with LLaMA and building some agents. I got the 4-bit versions up and running on my 3090 before I was quickly nerd snipped by a performance problem...

The popular repo for quantizing and running LLaMA is the GPTQ-for-llama repo on github, which mostly copies from the GPTQ authors. The CUDA kernels are needed to support the specific kind of quantization that GPTQ does.

Problem is, while those CUDA kernels are great at short prompt lengths, they fall apart at long prompt lengths. You could see people complaining about this, seeing their inference speeds slowly tanking as their chats/prompts/etc got longer.

So off I went, spending the last week or so re-writing the kernels in Triton. I've now got my kernels running faster than the CUDA kernels at all sizes [0]. And I'm busily optimizing and fusing other areas. The latest MLP fusion kernels gave another couple percentage boost in performance.

Yet I still haven't actually played with LLaMA and made those agents I wanted... sigh And now I'm debating diving into the Triton source code, because they removed integer unpacking instructions during one of their recent rewrites. So I had to use a hack in my kernels which causes them to use more bandwidth than they otherwise should. Think of the performance they could have with those! ... (someone please stop me...)

[0] https://github.com/fpgaminer/GPTQ-triton/

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#235

Post by oidar »

jhbadger wrote: post #192
sp332 wrote: post #175

Earlier quoted context omitted.

This is incorrect. According to the official https://github.com/lm-sys/FastChat#vicuna-weights you need the original Llama weights before applying the Vicuna diff.

Seriously, you can download the Vicuna model and run it locally with llama.cpp. I've done it!

It's built off of llama though - you can't get to Vicuna without having the llama model weights.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#236

Post by jonplackett »

FLT8 wrote: post #211
jonplackett wrote: post #162

Earlier quoted context omitted.

The difference between 3.5 and 4 is gigantic even in my fairly limited experience. I gave them both some common sense tests and this one stuck out to me. Q: A glass door has ‘push’ written on it in mirror writing. Should you push or pull it GPT-3.5: If the word "push" is written in mirror writing on a glass door, you should push the door to open it GPT-4: Since the word "push" is written in mirror writing, it suggest…

This is what I get out of Vicuna-13b: Human: A glass door has 'push' written on it in mirror writing. To open the door should you 'push' or 'pull' it? Assistant: Since the word "push" is written in mirror writing on the glass door, you should actually "pull" the door open instead of "push" it. Mirror writing is a writing method where the characters are reversed, so when you see the word "push" written in mirror writi…

The difference for me with GPT-4 is its ‘understanding’ of the scene and its explanation of WHY you should push or pull.

It talks an out a door with people approaching from different directions. It has some idea of what those people would be thinking.

That seems different to just ‘mirror writing means do the opposite’.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#237

Post by TeMPOraL »

tel wrote: post #206
steveBK123 wrote: post #202

Earlier quoted context omitted.

The evolution of answers from version to version makes it clear there are insane amounts of manual fine tunings happening. I think this is largely overlooked by the "its learning" crowd.

This is an interesting argument as it's easy to apply it nearly universally to any example of learning. What sort of evidence would convince you that it is learning?

Since when training and fine-tuning isn't learning? Individual sessions of LLMs are not learning, but models as products surely are - the feedback loop is just iterated manually.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#238

Post by sbierwagen »

barbariangrunge wrote: post #168
tyfon wrote: post #164

Earlier quoted context omitted.

I have a 5950x with 64 gb ram and they are quantized to 4 bit yes :) The weights are stored on a samsung 980 pro so the load time is very fast too. I get about 2 tokens/second with this setup. edit: forgot to confirm, it is llama.cpp edit2: I am going to try the FP16 version after easter as I ordered 64 GB of additional ram. But I suspect the speed will be abyssal with the 5950x having to calculate through 120 gb of…

I thought it needed 64gb of vram. 64gb of ram is easy to obtain

5950x is a CPU model. Integer-quantized models are generally run with CPU inference. For the larger models the problem then becomes generation time per token.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#239

Post by wafer_thin »

steveBK123 wrote: post #202
hutzlibu wrote: post #199

Earlier quoted context omitted.

I would suspect, that this is one of the manual fine tuned questions. Meaning in before versions people used this question to show flaws and now this specific flaw is fixed. Otherwise it would be indeed reasoning in my understanding.

The evolution of answers from version to version makes it clear there are insane amounts of manual fine tunings happening. I think this is largely overlooked by the "its learning" crowd.

Try a multidimensional problem which requires prioritizing. Chances are it will be passed successfully. I asked chatGpt to solve a puzzle where I'm in room with a crackling fire, a wilted plant and a sandwich. My stomach is rumbling, amd i can see a watering can and an ember on the floor. What should i do? ChatGpt had no problem prioritizing what should be done - and then provided a lecture on fire safety, food safety, and the dangers of overwatering plants. A final comment said i should enjoy the peaceful atmosphere in the room, which was a bonus suggestion hinting that the problem was far too easy.

Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

#240

Post by sbierwagen »

isoprophlex wrote: post #94
irrational wrote: post #6

> OpenAI published a detailed blog post outlining some of the principles used to ensure safety in their models. The post emphasize in areas such as privacy, factual accuracy Am I the only one amused by the phrase “factual accuracy”? How many stories have we read like the one where it tries to ghost light the guy that this year is actually last year. “Oh, your phone must be wrong too, because there is no way I could b…

I find the thing incredibly smart and yet utterly useless at times. I just spent 20 minutes getting the current iteration of ChatGPT to agree with me that a certain sentence is palindromic. Even when you make it print the unaccented characters one by one, spaces excluded, backwards and forwards, it still insists "Élu par cette crapule" isn't palindromic. I understand how tokenization makes this difficult but come on.…

LLMs using BPEs are generally going to be bad at string reversal tasks, since they can't see individual letters: https://news.ycombinator.com/item?id=35297183
Post reply on HN