Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta wants its open source AI model to be as capable as OpenAI’s best model

wsj.com

Post reply on HN
Print view View original

41–50 of 62 posts

Re: Meta wants its open source AI model to be as capable as OpenAI’s best model

#41

Post by someidiot123 »

kkielhofner wrote: post #22
rvz wrote: post #4

Anything to compete against the cloud AI models and instead run them locally for $0 for free on your own machine and anyone who is able to build such models has a moat such as Meta and is at the finish line in the AI race to zero. At this point it is unstoppable and $0 free local AI models will only accelerate and OpenAI and Google knows it.

It’s the early web all over again. Sun, DEC, Microsoft, etc just assumed they would use their cash and market presence to dominate the web up, down, left, and right. LAMP showed up and ate their lunch. Today open source is the underpinnings for basically every startup over the past 25 years and everything from Android to MacOS to every browser rendering engine. An exclusionary list would be easier. A 2008 study[0] fr…

Didn’t they say all this about crypto a few years ago?

Re: Meta wants its open source AI model to be as capable as OpenAI’s best model

#42

Post by PartiallyTyped »

labrador wrote: post #29
beebmam wrote: post #6

Earlier quoted context omitted.

Call me when it can compete with GPT-4. It continues to astound me.

GPT-4 answers are often kind of brilliant and surprising in a very interesting and pleasing way. I don't know why anybody would want to use a dumb AI if they can afford the best. Well worth the $20/month.

You can very much do even cheaper if you use the API directly.

Re: Meta wants its open source AI model to be as capable as OpenAI’s best model

#43

Post by labrador »

PartiallyTyped wrote: post #42
labrador wrote: post #29

Earlier quoted context omitted.

GPT-4 answers are often kind of brilliant and surprising in a very interesting and pleasing way. I don't know why anybody would want to use a dumb AI if they can afford the best. Well worth the $20/month.

You can very much do even cheaper if you use the API directly.

I don't have cable because I hate mainstream TV. ChatGPT is quite a bit cheaper than cable, so I can afford to be lazy about it. To be honest, I'm surprised so many are geeking out on the local models and not going for the most mysterious things you can do on advanced AI and then thinking about what the future holds. Some amazing things are coming down the pike. Who cares about the consumer app you think will make you rich?

Re: Meta wants its open source AI model to be as capable as OpenAI’s best model

#44

Post by BoorishBears »

cendyne wrote: post #39
ComputerGuru wrote: post #17

Earlier quoted context omitted.

What do you mean by “aligned?”

Alignment of large language models and machine learning is about shaping its behaviors to support the values of its maintainers and discourage behaviors that go against the same values. Like, you can't get a positive response by asking it to write a phishing letter for you, and it will refuse to divulge information it might know which is considered personally identifiable information. Unfortunately, the alignment app…

If you ask an LLM to complete a sentence like '[Insert name] stole the fruit (true/false):'

An aligned LLM will be biased towards refusing to answer at all with something like: "I can't tell you because I don't know them."

An "uncensored" LLM will very happily return or with a probability attached to each. Even OpenAI's GPT-3 does with a low enough temperature.

_

Of course, LLM attention doesn't work like that. The tokens are just a bag of numbers:

- The fact the name 'John' is mentioned in the Bible a lot affects the distribution when you ask if any John stole, because John is always [7554]

- The fact that 'Olf' is part of Adolf and Adolf Hitler is mentioned in a lot of negative sentences will drag the distribution, because 'Olf' is always [4024] and Adolf is always [324, 4024]

You could have asked something with no logical probability difference at all, like:

- 'The store attendant's name was [name], did the child in Long Island drop his ball (true/false):'

And unless you train the model to give you disclaimers it still follows the instruction faithfully and returns true/false with probabilities, demonstrating a deep regression in reasoning...

That's why for models past a certain size, alignment increases performance: https://arxiv.org/abs/2204.05862.

Re: Meta wants its open source AI model to be as capable as OpenAI’s best model

#45

Post by axitanull »

dartos wrote: post #38

Is it just me that’s bothered by people calling llama open source? It has a restrictive license.

Open sources can have a restrictive license, just because a project doesn't use WTFPL doesn't mean it's not open source.

Re: Meta wants its open source AI model to be as capable as OpenAI’s best model

#46

Post by grey8 »

AlexanderTheGr8 wrote: post #33

Does anyone know why Meta is open srcing Llama? The 2 explanations I have heard are "commoditize the complement" and "take advantage of public improvements of Llama". I have always found concepts such as "commoditize the complement" very interesting. For some reason, they seem like pseudo explanations of behaviours we see from companies. I am not entirely convinced that this was Meta's reasoning for releasing Llama.…

There is a podcast episode of Lex Fridman talking with Mark Zuckerberg, and in this episode Mark states that the reason Llama is (and will stay) open source is that that is the deal that Meta has with the engineers.

To get top engineers, there had to be some agreement, and this agreement apparently is that the engineers want their work open sourced.

Pretty cool, IMHO.

Re: Meta wants its open source AI model to be as capable as OpenAI’s best model

#47

Post by kromem »

flangola7 wrote: post #27
superkuh wrote: post #2

In terms of making something that could beat a turing test the 65B/70B llama1/2 already are better than openai's currently offered models like gpt3.5-turbo or even the currently available output from gpt3.5 text-davinci-003. They've been so heavily "aligned" that they're insufferable and "As a large language model," everything even given an extensive pre-prompt in text completion mode. It wasn't always this way. Back…

What are you asking it that hits the filter? I almost never see a refusal

It's not just about the filter.

This is a sleeper issue that's impacting all the fine tuned models right now.

They trained the models to complete massive amounts of human generated text, creating their foundational models.

Then they fine tuned the model to write like its an AI with no preferences or emotion -- which matched probably less than 0.1% of the training data the foundational model was trained on.

The end result is that the fine tuned models are better at a superficial level in response to broad, naive prompts. But much worse at the variety and quality of their foundational counterparts when paired with a well crafted text complete prompt.

As an example, try to get one of the fine tuned models to write marketing copy for a product. I guarantee about 25% of the responses will start with "Introducing..." (This isn't very good marketing copy.)

But try a modern foundational model and you'll only get that about 5% or less often.

They've inadvertently reduced the network search space by projecting our ideas of how AI should sound like after bad press when people were finding them too human.

But you can't have your cake and eat it too.

Either the models are very human in both how they sound and their capabilities, or they are much less than human in both outside of an overfitted scope of capabilities.

As much as people are blown away by the lobotomized GPT-4, I can't help but wonder just how good that foundational model could be.

So I'm quite excited at the prospect of Meta getting there and releasing the foundational version.

Re: Meta wants its open source AI model to be as capable as OpenAI’s best model

#48

Post by kkielhofner »

someidiot123 wrote: post #41
kkielhofner wrote: post #22

Earlier quoted context omitted.

It’s the early web all over again. Sun, DEC, Microsoft, etc just assumed they would use their cash and market presence to dominate the web up, down, left, and right. LAMP showed up and ate their lunch. Today open source is the underpinnings for basically every startup over the past 25 years and everything from Android to MacOS to every browser rendering engine. An exclusionary list would be easier. A 2008 study[0] fr…

Didn’t they say all this about crypto a few years ago?

Hah, sort of.

The difference is after 15 years the entire crypto ecosystem has somewhere in the range of 50-100m "users" worldwide (at best). ChatGPT alone has an estimated 100m monthly active users after less than a year...

You're correct in making the comparison but user adoption is the best metric to separate hype vs utility and AI is already absolutely blowing past crypto (obviously).

Re: Meta wants its open source AI model to be as capable as OpenAI’s best model

#49

Post by smoldesu »

labrador wrote: post #43
PartiallyTyped wrote: post #42

Earlier quoted context omitted.

You can very much do even cheaper if you use the API directly.

I don't have cable because I hate mainstream TV. ChatGPT is quite a bit cheaper than cable, so I can afford to be lazy about it. To be honest, I'm surprised so many are geeking out on the local models and not going for the most mysterious things you can do on advanced AI and then thinking about what the future holds. Some amazing things are coming down the pike. Who cares about the consumer app you think will make yo…

> not going for the most mysterious things you can do on advanced AI

Like what? In my experience, there are very few things that GPT-4 is definitively better at. It's good at explaining stuff, but not so good that I would deliberately avoid ChatGPT to use it. It's alright at coding, but code-optimized models regularly beat it in benchmarks. Plus, it's got the same lame filter and even an arbitrary request limit for paying customers.

I struggle to really imagine the "mysterious things you can do on advanced AI" that is impossible or impractical on local models. In my time using GPT-4, I was not really left wanting for more so much as I wanted it to stop saying "As an AI model..."

> Who cares about the consumer app you think will make you rich?

Evidently, people like you.

Re: Meta wants its open source AI model to be as capable as OpenAI’s best model

#50

Post by smoldesu »

wrsh07 wrote: post #37
AlexanderTheGr8 wrote: post #33

Does anyone know why Meta is open srcing Llama? The 2 explanations I have heard are "commoditize the complement" and "take advantage of public improvements of Llama". I have always found concepts such as "commoditize the complement" very interesting. For some reason, they seem like pseudo explanations of behaviours we see from companies. I am not entirely convinced that this was Meta's reasoning for releasing Llama.…

I strongly endorse the commodify complement hypothesis, but here are others: 1. The only one that matters - the controlling founder wants to 2. It's great marketing. Make Facebook engineering (which is talented) seem cool. Also let their talented engineering teams flex their muscles a bit. 3. Investing in GPUs is a decent hedge in case there is something world changing coming (imagine Instagram reels but 50% of the c…

Additionally, this is not an overnight decision for Facebook/Meta. A lot of people seem to be under the impression that this is a snap decision by them, but Facebook's AI research labs outdate and outspend most of FAANG's. Their only real competitor is Google, who practically rolled over dead once GGML/Pytorch cemented itself as the cutting-edge.

It's very possible that Llama had been ready for months, and Meta was internally unsure how to roll it out.

Post reply on HN