Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLaMA2 Chat 70B outperformed ChatGPT

tatsu-lab.github.io

Post reply on HN
Print view View original

41–50 of 135 posts

Re: LLaMA2 Chat 70B outperformed ChatGPT

#41

Post by beefield »

ramesh31 wrote: post #23
accrual wrote: post #8

Does this mean it may be possible to self-host a ChatGPT clone assuming you have a 70B model? I've used a 13B model with LLaMA1 and it's surprisingly good, but still nowhere near ChatGPT for coding questions.

>Does this mean it may be possible to self-host a ChatGPT clone assuming you have a 70B model? Not only possible but quite easy. Inference for 70B can be done with llama.cpp using CPU only, on any commodity hardware with >64GB of RAM

I have 64gb on my 5 year old thinkpad. What kind of performance (tokens per sec) I could expect on that nowadays for a 70B model?

Re: LLaMA2 Chat 70B outperformed ChatGPT

#42

Post by s17n »

lolinder wrote: post #37
letmevoteplease wrote: post #35

Earlier quoted context omitted.

There is one: "The agreement between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%). This means GPT-4’s judgments closely align with the majority of humans. We also show that GPT-4’s judgments may help humans make better judgments. During our data collection, when a human’s choice deviated from GPT-4, we presented GPT-4’s judgments to humans and ask if they are reasonable. De…

80% agreement is high, but the margins between models at the top are so low that even that remaining 20% could be enough to alter the final rankings, depending on which direction it errs.

But 80% agreement is the same as between humans

Re: LLaMA2 Chat 70B outperformed ChatGPT

#43

Post by lolinder »

s17n wrote: post #42
lolinder wrote: post #37

Earlier quoted context omitted.

80% agreement is high, but the margins between models at the top are so low that even that remaining 20% could be enough to alter the final rankings, depending on which direction it errs.

But 80% agreement is the same as between humans

True, but given that you're using one of the models to judge the others, it's likely that the cases of disagreement will tend to favor GPT-4. You would never use one of the competitors as a judge among humans.

Re: LLaMA2 Chat 70B outperformed ChatGPT

#44

Post by evilduck »

lolinder wrote: post #22
seydor wrote: post #12

this can apparently run on 48GB

When quantized to 4 bits, yes. You lose some quality by doing that, though, as compared to the full f16.

From what I've gathered when reading up on this topic, if RAM is your constraint the common thought has been that higher parameter models quantized down to smaller sizes will outperform lower parameter models running at higher quantization, i.e. it may still be preferable to use the 70B Llama model quantized to 4-bits than something like an unquantized f16 Falcon 40B or the "coming soon" f16 33B Llama2.

Re: LLaMA2 Chat 70B outperformed ChatGPT

#45

Post by lhl »

meepmorp wrote: post #21
lolinder wrote: post #18

Earlier quoted context omitted.

The advantage of LLaMA 2 is that a company can fine tune it on the knowledge that they actually care about and then run it on their own hardware without paying API fees or relying on an unstable dependency that's constantly being tweaked.

> without paying API fees or relying on an unstable dependency that's constantly being tweaked. and without handing a whole bunch of data to a 3rd party and hope they're securing it properly

Note, that just because you're hosting itself won't mean you're securing it properly... Here's a just published injection attack that only works on open source models (public model weights): https://twitter.com/random_walker/status/1683833600196714497

Re: LLaMA2 Chat 70B outperformed ChatGPT

#46

Post by sdenton4 »

letmevoteplease wrote: post #35
Jackson__ wrote: post #28

*When asked by GPT4 to compare the outputs. I'm a staunch believer that it would be foolish to rely on GPT4 for quality comparisons, and it has been mind boggling to see so many people do it and treat it as perfect proof of anything. It would be slightly more understandable if there was a study to see how human and gpt4 preferences compare, but I'm unaware of any such thing.

There is one: "The agreement between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%). This means GPT-4’s judgments closely align with the majority of humans. We also show that GPT-4’s judgments may help humans make better judgments. During our data collection, when a human’s choice deviated from GPT-4, we presented GPT-4’s judgments to humans and ask if they are reasonable. De…

That's helpful!

I've done a lot of work in audio synthesis, which is notoriously difficult measure. The gold-standard is human ratings of audio quality, but it is tough to design good tests (easy to fatigue raters) and the iteration time waiting for results is quite long.

Instead, there's now some projects which use neural networks trained on human ratings to predict audio quality, such as ViSQoL: https://github.com/google/visqol

This opens up fast iteration - scores going up generally corresponds to higher quality - followed by human testing at major milestones (eg, releasing a paper/model). VISQOL has a harder time comparing 'unrelated' models, IMO - ends up being not so great for comparison of different techniques, but excellent for measuring incremental improvement or catching regressions.

But, in the end, yes - you can use NN's to measure the quality of other NN's, so long as you're careful about it and make use of human raters from time to time as well.

The problem of test data getting into the training data seems to be an especially pernicious issue with LLM's, which isn't really arising in the audio synthesis space.

Re: LLaMA2 Chat 70B outperformed ChatGPT

#47

Post by sdenton4 »

lolinder wrote: post #37
letmevoteplease wrote: post #35

Earlier quoted context omitted.

There is one: "The agreement between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%). This means GPT-4’s judgments closely align with the majority of humans. We also show that GPT-4’s judgments may help humans make better judgments. During our data collection, when a human’s choice deviated from GPT-4, we presented GPT-4’s judgments to humans and ask if they are reasonable. De…

80% agreement is high, but the margins between models at the top are so low that even that remaining 20% could be enough to alter the final rankings, depending on which direction it errs.

If only there were some way to produce some kind of "interval" of scores where you were confident that the actual score sat, and then had some way of comparing these intervals between the different models...

Re: LLaMA2 Chat 70B outperformed ChatGPT

#49

Post by zo1 »

bazmattaz wrote: post #10
nmfisher wrote: post #3

I haven't had a chance to use the GPT-4 API yet - is it that much better than the GPT-4 available via ChatGPT? Or am I misunderstanding?

ChatGPT uses the GPT-4, but there are conspiracy theories circling that ChatGPT is neutered and thus not as good as GPT-4 through the API. The theory being that OpenAI are thottling the free version of GPT-4 (ChatGPT)

I use the GPT4 API and still think it got neutered since I first started using it.

Re: LLaMA2 Chat 70B outperformed ChatGPT

#50

Post by koalacola »

letmevoteplease wrote: post #35
Jackson__ wrote: post #28

*When asked by GPT4 to compare the outputs. I'm a staunch believer that it would be foolish to rely on GPT4 for quality comparisons, and it has been mind boggling to see so many people do it and treat it as perfect proof of anything. It would be slightly more understandable if there was a study to see how human and gpt4 preferences compare, but I'm unaware of any such thing.

There is one: "The agreement between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%). This means GPT-4’s judgments closely align with the majority of humans. We also show that GPT-4’s judgments may help humans make better judgments. During our data collection, when a human’s choice deviated from GPT-4, we presented GPT-4’s judgments to humans and ask if they are reasonable. De…

If it's trained by humans is it safe to assume that we'll get it so something crazy like 99% agreeable?
Post reply on HN