Live data from Hacker News

Llama 2

ai.meta.com

681–690 of 860 posts

Re: Llama 2

#681

Earlier quoted context omitted.

GPT models were internally available 6-12 months before they've seen public beta, of course OpenAI has more capable internal models.

There's no reason to believe this. The training time and cost is so substantial that they are almost certainly building their next release, but it isn't sitting there rotting.

Much of that training time is RLHF, the absence of which does not make the model less capable of carrying out useful tasks (indeed, in case of GPT-4, it actually made the model slightly less capable).

Re: Llama 2

#682

Earlier quoted context omitted.

> The brain is the result of maximizing biological objective functions. That's not how evolution works at all .

a mutation happens and if that mutation succeeds in ensuring survival, it stays and then spreads. Reproduce is a function evolution maximizes for. Not intentionally sure but that's irrelevant. The whole point of artificial neural networks is that they teach themselves. They get an answer wrong, numbers shift and if those numbers help the next instance they stay or shift as needed. There's no intentionality in the shi…

Evolution is not a mechanism that maximizes it is a set of interrelated elements that operate at entirely different levels (molecular, cellular, individual and species) to pass on those traits from one generation to another that result in (possible) change of future individuals within a population, which may affect the survival of those individuals increasing or decreasing the chances of passing those traits on to their (hopefully viable!) offspring. It does nothing to ensure survival of any particular individual, at best it may help a trait that has a function in survival to be passed on.

Mutations don't 'succeed' they get passed on or they don't, usually without an immediate effect on the individual, especially not in the individual where the mutation first takes place. But over a longer period mutations may result in a statistical advantage against some environmental factor (including predators) resulting in an increased chance of that particular mutation becoming more widespread. The result is possibly that that mutation ends up being carried by all individuals of the species but that takes a relatively long time and rarely results in 100% coverage in the first generations if at all.

ANN's do not necessarily 'teach themselves', there are supervised ways of using ANNs and there are unsupervised ways of using them.

> They get an answer wrong, numbers shift and if those numbers help the next instance they stay or shift as needed.

No, they optimize for the statistical likelihood of getting all of the answers right across the whole training set, and then we check if it worked by verification with held back data. So it is very well possible that a change in the weights negatively affects some outputs but positively affects others, if the change is a 'net positive' we keep it. To make matters more interesting every now and then we throw away all but a fraction of all the weights.

In the case of supervised learning we (humans) label the data and are the 'teachers' and in the case of unsupervised learning the ANN does its own labeling of clusters of related data (or, more accurately, clusters of data that seem to be related).

There are also intermediary versions where humans do some of the labeling to set up the initial clusters and then the mechanical labeling takes over to rapidly increase the size of the training set taking into account the possibility of getting some of the training data wrong but still coming out ahead across the board.

So yes, in ANNs there is a global maximization around a goal, and we set it up that way. Evolution doesn't have a particular goal, it is a result of a number of interrelated factors not an input and not something that was put together with a goal in mind.

Re: Llama 2

#683

One thing I haven't seen in the comments so far is that Llama 2 is tuned with RLHF [0], which the original Llama work wasn't. In addition to all the other "upgrades", seems like this will make it far easier to steer the model and get practical value. [0] Training Llama-2-chat: Llama 2 is pretrained using publicly available online data. An initial version of Llama-2-chat is then created through the use of supervised f…

On HF you'll see there's separate Llama-2-Xb and Llama-2-Xb-chat models, and more details on the model cards about -chat being the fine-tuned versions via SFT and RLHF.

Re: Llama 2

#684
post #460

Earlier quoted context omitted.

This sounds pretty good according to my memory. I did think it was first mentioned earlier than Path of Daggers. I don't remember it being used in The Last Battle but that was a pretty long chapter ...

It was used in The Last Battle throughout, but never focused on as a main set piece. It was just mentioned peripherally a couple of times that there was a large circle using the Bowl of the Winds to prevent catastrophic weather events from killing everyone during the battle/s.

Sounds about right now that you mention it. Time for a re read I guess.. :)

Re: Llama 2

#685
post #558

Earlier quoted context omitted.

Is it possible that some LLM’s are trained on these benchmarks? Which would mean they’re overfitting and are incorrectly ranked? Or am I misunderstanding these benchmarks?…

Presented with no comment :) https://twitter.com/chhillee/status/1635790330854526981?s=46...

[deleted]

Re: Llama 2

#686

Earlier quoted context omitted.

> Google's model is not as capable as llama-derived models, so I think they would actually benefit from this. Google's publically available model isn't as capable. But they certainly have models that are far better already in house.

Comments like this remind me of the old-timers from IBM saying "but wait, we invented the PC! and the cloud! and..." Gotta put products in the market, or it didn't happen...

OpenAI seemingly downgraded ChatGPT 4 due to the expense of running it for pro customers (unless you run it through the API).

Re: Llama 2

#687
post #684

Earlier quoted context omitted.

It was used in The Last Battle throughout, but never focused on as a main set piece. It was just mentioned peripherally a couple of times that there was a large circle using the Bowl of the Winds to prevent catastrophic weather events from killing everyone during the battle/s.

Sounds about right now that you mention it. Time for a re read I guess.. :)

As always :P me and my husband will be starting our first WoT reread together (I got him into the books after he loved the show), just as soon as we finish the current series we're on, which is Children of Time by Adrian Tchaikovsky. Absolutely fantastic series.

Re: Llama 2

#688

Earlier quoted context omitted.

Test leakage is not impossible for some benchmarks. But researchers try to avoid/mitigate that as much as possible for obvious reasons.

Given all of the times OpenAI has trained on peoples' examples of "bad" prompts, I am sure they are fine-tuning on these benchmarks. It's the natural thing to do if you are trying to position yourself as the "most accurate" AI.

[flagged]

Re: Llama 2

#689
post #520

Earlier quoted context omitted.

There's a few prompts that I use with every model to compare them. One of the simplest ones is: > When does the bowl of the winds get used in the wheel of time books? LLaMA2 fails pretty hard: > The Bowl of the Winds is a significant artifact in the Wheel of Time series by Robert Jordan. It is first introduced in the third book, "The Dragon Reborn," and plays a crucial role in the series throughout the rest of the bo…

I suppose that current LLMs are incapable of answering such questions by saying "I don't know". The have no notion of facts, or any other epistemic categories. They work basically by inventing a plausible-sounding continuation of a dialog, based on an extensive learning set. They will always find a plausible-sounding answer to a plausible-sounding question: so much learning material correlates to that. Before epistem…

In my experience, GPT-4 answers "I don't know" fairly frequently.

Re: Llama 2

#690

Earlier quoted context omitted.

Given all of the times OpenAI has trained on peoples' examples of "bad" prompts, I am sure they are fine-tuning on these benchmarks. It's the natural thing to do if you are trying to position yourself as the "most accurate" AI.

[flagged]

I am suggesting that it is only logical for a company whose main advertising comes from good benchmark numbers to play games with the benchmarks. In this case, I am suggesting that they run a fine-tuning/RL pass using benchmark scores as an objective function or using a training set that otherwise looks a lot like the benchmarks. Every single other company whose marketing depends on benchmarks does the analogue of this to some degree.

And we won't know for sure that they aren't doing this until they publicly disclose details about their model and training process (like every other research org does), allowing other researchers to run replication studies.

Also, I don't appreciate the ad hominems. Comments about some unrelated "conspiracy theorist" and "vaccine discourse" add nothing to the discussion.

Post reply on HN