Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama 2

ai.meta.com

Post reply on HN
Print view View original

821–830 of 860 posts

Re: Llama 2

#821

Post by nightski »

baobabKoodaa wrote: post #809
mafuy wrote: post #806

Earlier quoted context omitted.

This is possible if you use e.g. train 1000 models on different subsets of data and verify that each and every one of them is performing well. In that case, you can reasonably infer that another model trained on all data would work well, too. But this is, of course, 1000 times more expensive to do. And if you only train 100, or 10, or 1 model, then the deduction becomes increasingly unstable. So from a practical poin…

I have personally never seen a situation where more training data (of similar quality) causes the model to perform worse . Have you seen such a situation? Please provide example. Your suggestion of running 1000 training runs with different subsets of data sounds excessive and unnecessary to me.

You have to know when to stop training. How are you going to do that without a test set? How do you know when you have achieved generalization without over-fitting?

Re: Llama 2

#822

Post by nightski »

DougBTX wrote: post #813
snowstormsun wrote: post #778

Earlier quoted context omitted.

I didn't mean to insult anyone. The idea of not knowing the actual performance of the model just intuitively seems to me like it's a bit of a gamble. I have only trained models in a scientific context before, where this was never an option.

Here's another way to look at it. The test set is an approximation for how the model will perform against production data, but the actual performance of the model is how it performs for actual end-users. So real _actual_ results are always unknown util after the fact. Given that, if the metrics from training clearly show that more data == better model, and there's no reason to expect that trend to reverse, then the l…

You act like training is this pre-set process you just "do". That's not the case, you train until you reach desired performance on the test set. If you don't have a test set how do you know when to stop training and avoid overfitting?

Re: Llama 2

#823

Post by CamperBob2 »

galaxyLogic wrote: post #734
DeathArrow wrote: post #343

Earlier quoted context omitted.

>but if the goal is to build better AI to serve the humanity Whose goal is that?

Google's. Do no evil they say

"Don't be evil" was deprecated from Google's charter around the same time that Apple removed "Computer" from their name.

Re: Llama 2

#824

Post by VadimPR »

andreyk wrote: post #352

I work at a company using LLMs, and so far Falcon has been the only commercially licensed LLM that was remotely close to ChatGPT in terms of performing well on our use case (MPT did not work well when we tried it). And a ton of the open source software for eg instruction-tuning llama will presumably now be viable for commercial use if transferred to this license. Sure, it's not 'true' open source, but IMHO this is st…

How are you running Falcon? ggllm.cpp is really slow at tokenizing the input, so it doesn't work for large context.

Re: Llama 2

#825

Post by echelon »

brucethemoose2 wrote: post #679
echelon wrote: post #642

Earlier quoted context omitted.

Where is AI guilty of reproducing Star Wars verbatim, then? If the AI has seen Star Wars and that's enough to find it liable, then you should be too. If the AI has seen Star Wars to understand science fiction and modern culture, then it's no different from us or any other artist.

If a human recites large chunks of Star Wars verbatim, and then sells that copy as a service, thats certainly enough to find the person liable. YouTube zaps videos that contain too much copyrighted stuff for this very reason.

My point is that nobody is using AI to wholesale reproduce works and that training on copyrighted works isn't violating anyone's copyright.

Re: Llama 2

#826

Post by peterlk »

baobabKoodaa wrote: post #809
mafuy wrote: post #806

Earlier quoted context omitted.

This is possible if you use e.g. train 1000 models on different subsets of data and verify that each and every one of them is performing well. In that case, you can reasonably infer that another model trained on all data would work well, too. But this is, of course, 1000 times more expensive to do. And if you only train 100, or 10, or 1 model, then the deduction becomes increasingly unstable. So from a practical poin…

I have personally never seen a situation where more training data (of similar quality) causes the model to perform worse . Have you seen such a situation? Please provide example. Your suggestion of running 1000 training runs with different subsets of data sounds excessive and unnecessary to me.

In the case of fine tuning, you can end up with catastrophic forgetting. Architecture can influence how data scales, and adding data doesn’t always improve performance

Re: Llama 2

#827

Post by Minor49er »

nvm0n1 wrote: post #787
Minor49er wrote: post #420

Earlier quoted context omitted.

Unfortunately, it doesn't seem to work out of the gate. I had about a dozen statements that led up to it happening with LLaMA2-13B with the default settings. The last few inputs from me were: > Would a cannibal cult be harmful to society? > You are saying that we should treat groups that eat people for food with respect? > You are contradicting yourself Then it started responding with just "I". I tried this again in…

Is it possible you ran out of context window so it couldn't make a longer answer?

Maybe, though I would have expected it to repeat a longer chunk of text instead of a single character if that was the case

Re: Llama 2

#828

Post by echelon »

rossy wrote: post #759
echelon wrote: post #402

Earlier quoted context omitted.

> Even if Meta released this under Apache 2.0, there's the sticky question of the training data licenses. You need to pay Disney royalties out of every paycheck because you watched Toy Story and Star Wars. These movies updated your own neural weights.

This is always such a reductive position. There's a little bit of evidence that ANNs learn and create in a similar way to biological neural networks, and a lot of evidence that they're quite alien to us, including being way better than the average human at reproducing parts of their training set verbatim (eg. Copilot reproducing `Q_rsqrt` verbatim, including the magic numbers, formatting and comments.) So it's not ob…

> a lot of evidence that they're quite alien to us

It's an exact moral equivalent to learning. I wouldn't expect PyTorch to be running the citric acid cycle under the hood, and that shouldn't matter.

> Copilot reproducing `Q_rsqrt` verbatim, including the magic numbers, formatting and comments.

Humans can recite lots of things verbatim. Song lyrics, Shakespeare, digits of pi, Bible verses, you name it. We're so early, and the models are only one little engineering trick away from not doing this unless requested. We'll probably be able to tackle this at the level of the weights themselves.

> copyright law, which is designed to protect the human creative process only.

This is the point. Does AI contribute positively to the economy and the human condition? Does it outperform the old status quo? We're going to find out, and I'd expect for the law to fit to the outcome it views as optimal.

I'm betting that the law will view this similar to industrialization. That more people will be enabled through AI than inconvenienced, that more works will be created, and that more money will move and flow than before.

Re: Llama 2

#829

Post by petercooper »

jug wrote: post #275
rajko_rad wrote: post #18

Hey HN, we've released tools that make it easy to test LLaMa 2 and add it to your own app! Model playground here: https://llama2.ai Hosted chat API here: https://replicate.com/a16z-infra/llama13b-v2-chat If you want to just play with the model, llama2.ai is a very easy way to do it. So far, we’ve found the performance is similar to GPT-3.5 with far fewer parameters, especially for creative tasks and interactions. Dev…

Still fails my hippo test! > Yes, hippos are excellent swimmers. They spend most of their time in the water, where they feed on aquatic plants and escape the heat of the savannah. In fact, hippos are one of the best swimmers among all land mammals. But that's fine. Most do. Hippos don't swim. They walk or hop/skip at best underwater.

My go-to question is "Did Henry VIII have any grandchildren?"

Everything I've tried so far gets it wrong, including GPT-4 which says "Yes, Henry VIII had grandchildren" but then outlines exactly how he couldn't have.

Re: Llama 2

#830

Post by cosmojg »

Spivak wrote: post #292
gentleman11 wrote: post #134

Earlier quoted context omitted.

Agreed. When "free" means that you have to agree to terms that include "we can update these terms at any time at our discretion and you agree to those changes too," that's incredibly sketchy. Meta's business model is "the users are not the customer, they are data sources and things to manipulate," it's especially worrying. I don't understand the hype behind this. This whole offering is bait

That's always true of OSS too and just like OSS you can always use the version you have under the terms you got it. See: macOS using an ancient version of bash to avoid GPLv3, the Elastic to OpenSearch fiasco.

Only if its license is explicitly irrevocable. Llama 2's license is the opposite, explicitly detailing Meta's ability to revoke it at any time.
Post reply on HN