Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLaMA2 Chat 70B outperformed ChatGPT

tatsu-lab.github.io

Post reply on HN
Print view View original

121–130 of 135 posts

Re: LLaMA2 Chat 70B outperformed ChatGPT

#121

Post by swyx »

sytelus wrote: post #62
alecco wrote: post #13

Better evaluation paints a bit different picture: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... *FreeWilly2 is a Llama2 70B model finetuned on an Orca style Dataset EDIT: actually, impressive: FreeWilly2 GPT-3.5 GPT-4 ARC 71.1 85.2 96.3 HellaSwag 86.4 85.5 95.3 MMLU 68.8 70.0 86.4 TruthfulQA 59.4 47.0 59.0 So reasoning (ARC) is lagging behind, but the other evaluations are at GPT-3.5 level and closi…

LLaMA2 is far and away from GPT 3.5. Just look at HumanEval and other code generation metrics. All these GPT-4 based "chat evals" are extremely misleading and people should take it with a bag of salt.

many usecases that arent code, + people can finetune llama2 with code. its a great start for a free model.

Re: LLaMA2 Chat 70B outperformed ChatGPT

#122

Post by dsabanin »

drew-y wrote: post #85
europeanNyan wrote: post #65

There is a cool website where you can blind judge the outputs from LLaMa 2 vs ChatGPT-3.5: https://llmboxing.com/ Surprisingly, LLaMa 2 won 5-0 for me.

I got the opposite result. ChatGPT-3.5 won 5-0 for me. For me, LLaMa 2 gave longer answers that sometimes strayed away from the original question. They both gave great answers overall though.

Same for me. Interesting..

Re: LLaMA2 Chat 70B outperformed ChatGPT

#123

Post by wahnfrieden »

hobomatic wrote: post #120
wahnfrieden wrote: post #93

Earlier quoted context omitted.

What’s the balance between, say, opinions that trans people should be exterminated vs not? What’s the balance between Ukraine sovereignty vs Russian occupation? Etc

It would involve serious consideration of the opinions you don't agree with, rather than just qualifying them in the most hyperbolic and dismissive way possible. Hopefully ai is better capable of this sort of reasoning than people are.

hi

Re: LLaMA2 Chat 70B outperformed ChatGPT

#124

Post by Tostino »

agravier wrote: post #116
Tostino wrote: post #64

Earlier quoted context omitted.

Youtube transcripts. It only works with single-person channels at the moment, as I haven't worked on disambiguating multiple speakers. They are very messy if they are just an auto transcription from Google. Practically unusable in most cases. So first step in the pipeline is cleaning up the transcripts for incorrectly transcribed words or sentences. Using the context of the rest of the transcript, it is able to fix t…

How do you represent knowledge in your knowledge graph? Do you use an existing open source ontology?

No, I just used YAML to represent it.

Here is an example: https://gist.github.com/Tostino/f6f19e88e39176452c1a765cb7c2...

Here is the transcript that I created that knowledge graph from, and then annotated with the knowledge graph for training purposes: https://gist.github.com/Tostino/e64524437848fbb3aebe52056df8...

Edit: I am using symbolic IDs intentionally. Reason for that was this paper: https://ai.googleblog.com/2023/07/symbol-tuning-improves-in-...

Re: LLaMA2 Chat 70B outperformed ChatGPT

#126

Post by huggingmouth »

lhl wrote: post #45
meepmorp wrote: post #21

Earlier quoted context omitted.

> without paying API fees or relying on an unstable dependency that's constantly being tweaked. and without handing a whole bunch of data to a 3rd party and hope they're securing it properly

Note, that just because you're hosting itself won't mean you're securing it properly... Here's a just published injection attack that only works on open source models (public model weights): https://twitter.com/random_walker/status/1683833600196714497

There's a world of difference between visiting links hallucinated by an unreliable ai and having a third party store (and possibly sell) every single thing you say to an ai and tying it to your identity, forever.

Also there is nothing about that attack that makes it iinherently only applicable to self hosted models.

Re: LLaMA2 Chat 70B outperformed ChatGPT

#127

Post by sdenton4 »

2c2c2c wrote: post #108
sdenton4 wrote: post #46

Earlier quoted context omitted.

That's helpful! I've done a lot of work in audio synthesis, which is notoriously difficult measure. The gold-standard is human ratings of audio quality, but it is tough to design good tests (easy to fatigue raters) and the iteration time waiting for results is quite long. Instead, there's now some projects which use neural networks trained on human ratings to predict audio quality, such as ViSQoL: https://github.com/…

just curious, are there any open models doing the opposite of audio synthesis? As in able to generate the stems for a song?

Ha, I was working on neutral voice compression, which consists of an encoder which creates what the kids are calling tokens these days, and a decoder which synthesizes speech from the tokens.

AudioLM puts a language model on top of the compression tokens, and thus can generate speech or other audio. There's piles of recent papers pushing that approach into music generation. Mulan is a name that comes to mind.

Or maybe your interested in audio separation to get at the isolated instruments? There's lots of great work on that, as well. Like MixIT, which is an unsupervised audio separation system.

Re: LLaMA2 Chat 70B outperformed ChatGPT

#128

Post by bottlepalm »

beepbooptheory wrote: post #113
bottlepalm wrote: post #103

Earlier quoted context omitted.

I asked it where moral relativism fit in with philosophy and it came back with this Philosophy Ethics/Moral Philosophy Meta-Ethics: The study of moral thought, language, and properties Moral Realism: Belief that there are objective moral facts Moral Anti-Realism: Denial of the existence of objective moral facts Moral Relativism: The belief that moral judgments are true or false only relative to some particular standp…

Haha I think it's fine. I think its kinda cheeky answering you so literally, giving it an actual place to fit into :). I don't doubt it can do, like, Wikipedia type classification ok, but that's not like really getting to the substance of anything! And, either way, its not like there is one decided-upon hierarchy of concepts like this people consciously work within. This is a fine picture to some, but others might co…

Yea I don’t think it was trained on any hierarchy in particular, ChatGPT literally came up with this on the fly. There’s even more to it I didn’t include to keep the post small. Just really impressive to have a conversation with it that goes all sorts of places. I don’t know any philosophy professors so this is the next best thing.

Re: LLaMA2 Chat 70B outperformed ChatGPT

#129

Post by lhl »

huggingmouth wrote: post #126
lhl wrote: post #45

Earlier quoted context omitted.

Note, that just because you're hosting itself won't mean you're securing it properly... Here's a just published injection attack that only works on open source models (public model weights): https://twitter.com/random_walker/status/1683833600196714497

There's a world of difference between visiting links hallucinated by an unreliable ai and having a third party store (and possibly sell) every single thing you say to an ai and tying it to your identity, forever. Also there is nothing about that attack that makes it iinherently only applicable to self hosted models.

That specific attack requires adversarial inputs crafted crafted against gradients so only works against open models (requires known model weights). There are dangers that include leaking PII from your current context, but also worse if you are using the model with RAG or with other types of system access so I don't think it's as innocuous as you are assuming.

Re: LLaMA2 Chat 70B outperformed ChatGPT

#130

Post by drittich »

shanusmagnus wrote: post #74
bostonsre wrote: post #61

Earlier quoted context omitted.

Is perfect agreement possible? And what is the definition of agreement? Humans don't agree about much.. are we saying agreement means it matches the truth after intensive investigation by humans?

Ugh, when I was doing my PhD work we were studying creativity in an experiment, and we needed an assessment for how creative different solutions were, and trying to get inter-rater reliability on this quite simple thing was just agonizing. I wound up abandoning the experiment because getting enough reliability would have required screwing down the standards so tightly that it would have ruined the underlying point of…

It's surprising to me that you describe creativity as a simple thing. How were you defining and measuring it?
Post reply on HN