Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama 2

ai.meta.com

Post reply on HN
Print view View original

511–520 of 860 posts

Re: Llama 2

#511

Post by wizofaus »

TrevorJ wrote: post #477
jug wrote: post #275

Earlier quoted context omitted.

Still fails my hippo test! > Yes, hippos are excellent swimmers. They spend most of their time in the water, where they feed on aquatic plants and escape the heat of the savannah. In fact, hippos are one of the best swimmers among all land mammals. But that's fine. Most do. Hippos don't swim. They walk or hop/skip at best underwater.

I suspect LLM's don't know this because most people don't know this, so the training data is largely absent this fact.

Perhaps but LLMs have been trained on information that most people haven't. Most of the stuff ChatGPT is correct about are things "most people" wouldn't know the answer to anyway.

Re: Llama 2

#512

Post by nine_k »

jug wrote: post #275
rajko_rad wrote: post #18

Hey HN, we've released tools that make it easy to test LLaMa 2 and add it to your own app! Model playground here: https://llama2.ai Hosted chat API here: https://replicate.com/a16z-infra/llama13b-v2-chat If you want to just play with the model, llama2.ai is a very easy way to do it. So far, we’ve found the performance is similar to GPT-3.5 with far fewer parameters, especially for creative tasks and interactions. Dev…

Still fails my hippo test! > Yes, hippos are excellent swimmers. They spend most of their time in the water, where they feed on aquatic plants and escape the heat of the savannah. In fact, hippos are one of the best swimmers among all land mammals. But that's fine. Most do. Hippos don't swim. They walk or hop/skip at best underwater.

I think this is unfair. Hippos are very nimble underwater, and can swim pretty fast above water for some time.

Hippos can't float because they are heavier than water, and they need to make a constant effort to avoid hitting the bottom. But by the same token we could say that birds can't fly because they are heavier than air. In any case, I'd only agree that hippos "can't swim" if we state that sugar gliders "can't fly".

Re: Llama 2

#513

Post by chaxor »

GreedClarifies wrote: post #108
chaxor wrote: post #60

Earlier quoted context omitted.

Google has far better models than llama based models. They just simply don't put them facing the public. It is pretty ridiculous that they essentially just set a marketing team with no programming experience to write Bard, but that shouldn't fool anyone into believing they don't have capable models in Google. If Deepmind were to actually provide what they have in some usable form, it would likely be quite good. Despi…

Hard disagree. Google has made it plainly clear that they don't have anything useable in this space. Bard scores below all other commercial model. Google is getting the asses handed to them, badly. I figured that the code red would whip them into shape but the rot runs deep.

It seems you didn't quite hear the argument. I agree with you that the models Google has released to the public are absolutely worthless. That certainly does not mean they don't have extremely performant models at all however.

If you actually have worked in the area of NLP for about 10 years, you would recognize how the work from Deepmind is much more novel and innovative than other groups. OpenAI certainly has great public facing services, and Meta should be congratulated for releasing these models (although I would still prefer the Galactica training data), but academically Deepmind is one of the best groups around.

Re: Llama 2

#514

Post by zamadatix »

alexeldeib wrote: post #372
zamadatix wrote: post #363

Earlier quoted context omitted.

To be fair on that, both the US and EU governments launched antitrust cases around that with the US case narrowly avoiding having the company split up and the EU ruling resulting in requirements the browser be decoupled, followed by half a billion in fines for not doing so well enough. Not that the two situations are anything alike, but a "and look what happened with that" argument hardly points away from valid antit…

I think you and parent/GP all agree? A thing can be anti competitive, and a strategic failure.

If that's what the parent and GP are saying then we definitely don't agree. In my mind, it was anticompetitive and a rousing success. Microsoft managed to fully execute the extend and extinguish phases to then hold a stranglehold on the web for roughly a decade at a cost of less than a billion dollars. Anticompetitive measures kept it from being worse, but it was far from a bad outcome for Microsoft either.

Re: Llama 2

#515

Post by lappa »

Here are some benchmarks, excellent to see that an open model is approaching (and in some areas surpassing) GPT-3.5!

AI2 Reasoning Challenge (25-shot) - a set of grade-school science questions.

- Llama 1 (llama-65b): 57.6

- LLama 2 (llama-2-70b-chat-hf): 64.6

- GPT-3.5: 85.2

- GPT-4: 96.3

HellaSwag (10-shot) - a test of commonsense inference, which is easy for humans (~95%) but challenging for SOTA models.

- Llama 1: 84.3

- LLama 2: 85.9

- GPT-3.5: 85.3

- GPT-4: 95.3

MMLU (5-shot) - a test to measure a text model’s multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more.

- Llama 1: 63.4

- LLama 2: 63.9

- GPT-3.5: 70.0

- GPT-4: 86.4

TruthfulQA (0-shot) - a test to measure a model’s propensity to reproduce falsehoods commonly found online. Note: TruthfulQA in the Harness is actually a minima a 6-shots task, as it is prepended by 6 examples systematically, even when launched using 0 for the number of few-shot examples.

- Llama 1: 43.0

- LLama 2: 52.8

- GPT-3.5: 47.0

- GPT-4: 59.0

[0] https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... [1] https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

Re: Llama 2

#516

Post by unshavedyak »

ipaddr wrote: post #335
freedomben wrote: post #264

Earlier quoted context omitted.

I don't think that's even possible, but if it was it would be a disaster because humans don't work that way. We respond to incentive. When giving to charity, the incentive can be as simple as "I feel good" but it's still an incentive.

Some do what's right even if it doesn't feel good. The best charity can be painful.

What? Pain has nothing to do with this. The positives outweigh the negatives by whatever criteria the giver uses to decide to do the act of giving.

This is always true. No? Even self sacrifice, such as jumping in front of a bus, you deem to be the right thing - you chose an action because you believed it was a positive, for you, action. Just because you die doesn't mean you didn't still felt it was right, in that moment.

If you do something you do not believe is a positive it changes from charity to masochism. You're doing it to punish yourself. If you're not punishing yourself, you're gaining something (or net neutral, i suppose, but that's difficult to define).

edit: Though arguably even punishing yourself is self interest, as if you're punishing yourself it's because you want to.

Re: Llama 2

#517

Post by Patrick_Devine »

simonw wrote: post #499
jmorgan wrote: post #492

Earlier quoted context omitted.

If you want to try running Llama 2 locally, you can use https://github.com/jmorganca/ollama To run Llama 2 with it: ollama run llama2

When you run that command, where does it download the model weights from?

They're stored in a registry (based on Docker distribution) running on Cloudflare. The model gets broken up into layers, so if you want to create new prompts or parameters, you can create something called a Modelfile (similar to a Dockerfile) and just point it at the existing model.

If you want to create a new Modelfile, you can create something like:

  FROM llama2
  PARAMETER temperature 0.4
  PROMPT """

  >
  """
And then:

  ollama create  -f 
  ollama run

Re: Llama 2

#518

Post by sltkr »

mkl wrote: post #506
CSMastermind wrote: post #332

Earlier quoted context omitted.

There's a few prompts that I use with every model to compare them. One of the simplest ones is: > When does the bowl of the winds get used in the wheel of time books? LLaMA2 fails pretty hard: > The Bowl of the Winds is a significant artifact in the Wheel of Time series by Robert Jordan. It is first introduced in the third book, "The Dragon Reborn," and plays a crucial role in the series throughout the rest of the bo…

> Here LLaMA2 also fails pretty hard, though I thought this follow up response was pretty funny: > > The function would return true for 'IIIIII' because it contains the Roman numeral 'IV'. That's arguably correct. 'IIII' is a valid Roman numeral representation of 4 [1], and the string 'IIIIII' does contain 'IIII'. [1] https://en.wikipedia.org/wiki/Roman_numerals#Other_additive_...

Since you're being pedantic my reply is going to be equally pedantic: no, this is not correct if you understand the difference between numerals and numbers.

A numeral is a written way of denoting a number. So while the string "IIIIIIII..." arguably contains a Roman numeral denoting the number 4 as a substring (if you accept "IIII" as a Roman numeral), it still does not contain the Roman numeral "IV" as a substring.

Or phrased differently, by your logic you might as well say that "IIIIIIII..." contains the Arabic numeral "4". It doesn't.

Re: Llama 2

#519

Post by visarga »

brucethemoose2 wrote: post #249
joshhart wrote: post #206

From a modeling perspective, I am impressed with the effects of training on 2T tokens rather than 1T. Seems like this was able to get LLAMA v2 7b param models equivalent to LLAMA v1's 13b performance, and the 13b similar to 30b. I wonder how far this can be scaled up - if it can, we can get powerful models on consumer GPUs that are easy to fine tune with QLORA. A RTX 4090 can serve an 8-bit quantized 13b parameter mo…

At some point, higher quality tokens will be far more important than more tokens. No telling how much junk is in that 2T. But I wonder if data augmentations could help? For instance, ask LLaMA 70B to reword everything in a dataset, and you can train over the same data multiple times without repeats.

A great idea. If we are at it, why don't we search all topics and then summarise with a LLM? It would be like an AI made wikipedia 1000x times larger indexing all things, concepts and events, or a super knowledge graph. It would create a lot of training data, and maybe add a bit of introspection to the model - it explicitly knows what it knows. Could help reduce hallucinations, learn attribution, ability to recognise copyrighted content, and fact checking.

Re: Llama 2

#520

Post by nine_k »

CSMastermind wrote: post #332
jug wrote: post #275

Earlier quoted context omitted.

Still fails my hippo test! > Yes, hippos are excellent swimmers. They spend most of their time in the water, where they feed on aquatic plants and escape the heat of the savannah. In fact, hippos are one of the best swimmers among all land mammals. But that's fine. Most do. Hippos don't swim. They walk or hop/skip at best underwater.

There's a few prompts that I use with every model to compare them. One of the simplest ones is: > When does the bowl of the winds get used in the wheel of time books? LLaMA2 fails pretty hard: > The Bowl of the Winds is a significant artifact in the Wheel of Time series by Robert Jordan. It is first introduced in the third book, "The Dragon Reborn," and plays a crucial role in the series throughout the rest of the bo…

I suppose that current LLMs are incapable of answering such questions by saying "I don't know". The have no notion of facts, or any other epistemic categories.

They work basically by inventing a plausible-sounding continuation of a dialog, based on an extensive learning set. They will always find a plausible-sounding answer to a plausible-sounding question: so much learning material correlates to that.

Before epistemology is introduced explicitly into their architecture, language models will remain literary devices, so to say, unable to tell "truth" from "fiction". All they learn is basically "fiction", without a way to compare to any "facts", or the notion of "facts" or "logic".

Post reply on HN