Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLaMA: A foundational, 65B-parameter large language model

ai.facebook.com

Post reply on HN
Print view View original

201–209 of 209 posts

Re: LLaMA: A foundational, 65B-parameter large language model

#201

Post by hansvm »

akomtu wrote: post #200
hansvm wrote: post #199

Earlier quoted context omitted.

Ah, so less a hash table and more vanilla KNN? Edit: I missed this on the first pass, but I'm totally lost as to where 1.5T comes from. Even if you only have two tokens there are vastly more 8192-length subsequences than that (something like 2^8151.5 times more), and if we're just trying to replicate the same space as something like GPT3.5 or LLaMA then you only get on the order of 0.065T to 0.175T entries to play wi…

k-nearest neighbors? Sort of, but I'd rather describe it as a geospatial map in many dimensions.

There are lots of interpretations. I actually like KNN for a lot of tasks. My gut says that it still wouldn't perform well here (and for the record, there are efficient data structures for the idea you're describing unless you have some nonstandard modifications, so "arranging the data efficiently to allow fast lookups" is definitely not the core problem), but I admittedly don't have proof of that yet.

For some intuition, imagine the following tasks:

> Repeat the following phrase exactly twice: "sdflhasdflhasdf"

> Repeat the following phrase exactly twice: "sdflhasdflhasdg"

Your fuzzy dictionary or geospatial map can't possibly have enough keys to distinguish the requests (or if it distinguishes those, you can adversarially select different keyboard mashes), and so the result, no matter what it is, would have the same probability distribution for both prompts. Since the desired results are different, at least one of those would be have some unavoidable wrongness.

The GPT family, on the other hand, has few issues with random phrase duplication since positional information is something it explicitly considers and is capable of prioritizing over other token information.

Re: LLaMA: A foundational, 65B-parameter large language model

#202

Post by sandGorgon »

cubefox wrote: post #185
sandGorgon wrote: post #178

Earlier quoted context omitted.

hey thanks for your reply. umm...so does OpenAI. In fact this is OpenAI discovery from [1]: > Convergence is inefficient: When working within a fixed compute budget C but without any other restric- tions on the model size N or available data D, we attain optimal performance by training very large models and stopping significantly short of convergence (see Figure 3). Maximally compute-efficient training would therefor…

This is the old scaling laws paper. The scaling laws in it turned out to be wrong and superseded by the Chinchilla DeepMind paper: https://arxiv.org/abs/2203.15556

hi again - genuinely trying to learn here. The Chinchilla paper is a COMPETING thesis right ? the OpenAI thesis hasnt changed or superseded here.

Re: LLaMA: A foundational, 65B-parameter large language model

#203

Post by PoignardAzur »

theRealMe wrote: post #46
freeqaz wrote: post #17

Earlier quoted context omitted.

How did Microsoft sour developers with Copilot? I know dozens of people that pay for it (including myself) and I feel like it is widely regarded as a "no brainer" for the price that it's offered at. Please help me understand!

They didn’t. There is a small group of people that are always looking for the latest reason to be outraged and to point at any of the big tech companies and go “aha! They are evil!” Copilot’s ai was trained on GitHub projects and so these people are taking their turns clutching their pearls inside of their little bubble. I’d bet that more than 95% of devs haven’t even heard of this “controversy” and even if they did,…

I'm not so sure.

I do think the controversy is stupid, but inside my own company, we significantly delayed migrating some projects to Github because people were concerned that the way Microsoft handled Copilot meant that Github wasn't a safe long-term host for an open-source project (and yes, I'm aware of all the reason that's irrational).

Even if the people angry about Copilot are a minority, it might still have a bad move. Trust accumulates slowly over years, but mistrusts builds up over only a few events. People are still remembering Microsoft's anticompetitive practices from 20 years ago. The mistakes it makes now might stick for a long time.

Re: LLaMA: A foundational, 65B-parameter large language model

#204

Post by PoignardAzur »

sandkoan wrote: post #19
freeqaz wrote: post #17

Earlier quoted context omitted.

How did Microsoft sour developers with Copilot? I know dozens of people that pay for it (including myself) and I feel like it is widely regarded as a "no brainer" for the price that it's offered at. Please help me understand!

Presumably, because they trained Copilot on billions of lines of, often licensed, code (without permission), that Copilot has a tendency to regurgitate verbatim, without said license.

> that Copilot has a tendency to regurgitate verbatim, without said license.

A "tendency" is overstating it. I'm not aware of any example that would have been likely to occur if the author wasn't specifically trying to get the regurgitated code.

Re: LLaMA: A foundational, 65B-parameter large language model

#205

Post by akomtu »

hansvm wrote: post #201
akomtu wrote: post #200

Earlier quoted context omitted.

k-nearest neighbors? Sort of, but I'd rather describe it as a geospatial map in many dimensions.

There are lots of interpretations. I actually like KNN for a lot of tasks. My gut says that it still wouldn't perform well here (and for the record, there are efficient data structures for the idea you're describing unless you have some nonstandard modifications, so "arranging the data efficiently to allow fast lookups" is definitely not the core problem), but I admittedly don't have proof of that yet. For some intui…

Indeed, that's good counterexample.

Re: LLaMA: A foundational, 65B-parameter large language model

#206

Post by nathanh4903 »

adeon wrote: post #59

Has anyone had success being approved to download weights even if you are just a hobbyist with a big GPU? (I'm here about to begrudgingly fill out the Google Form) Asking in general, not just for this particular model. I've been following these language models and so far went through approval once for a English/Chinese GLM-130B model and it took only ~30 minutes to get approved even though I am a complete nobody.

i got in after 2 days and im just an undergrad with some experience tangentially related to machine learning. Too bad i only found out the hardware requirement after downloading 13GB of weights

Re: LLaMA: A foundational, 65B-parameter large language model

#207

Post by bobobobobobobo »

iandanforth wrote: post #95

"we are publicly releasing LLaMA" "Access to the model will be granted on a case-by-case basis to academic researchers" They keep saying the word 'release' but I don't think they know what that word means. There are perfectly good words in the English language to describe this situation without abusing "release." They "will begin to grant access to a select few". Nothing about that releases the model, or their contro…

[flagged]

Re: LLaMA: A foundational, 65B-parameter large language model

#208

Post by webmaven »

invig wrote: post #130
webmaven wrote: post #90

Earlier quoted context omitted.

> Facebook has no real way to monetize this (eg they won’t release an api a la OpenAI and they don’t own a search engine). Since they can’t monetize it… why not provide a bit of kindling to lower the barrier for everyone else to compete with your competitors. This strategy is called “commodize your complement”. Your analysis is good, but that isn't what "commoditize your complement" means. Strictly speaking, a search…

I don't think that's the best interpretation of complement. Think of complement as "is your users experience benefited by the existence of the other thing". I think that Search qualifies as something that enhances the Facebook users experience (e.g. my cousin mentioned thing that they like doing today, how do I find more about thing?)

Given FBs optimization of the feed for dopamine hits, I don't think search across posts is a high priority (yes, it is table stakes, but whatever they currently use seems to be commoditized enough for their purposes). Group and page recommendations and discovery is another matter, perhaps an LLM may help make that more engaging. It might also be useful for helping users (and brands) moderate their groups and so on. These are also complements, but they aren't external ones.
Post reply on HN