Live data from Hacker News

LLaMA: A foundational, 65B-parameter large language model

ai.facebook.com

81–90 of 209 posts

Re: LLaMA: A foundational, 65B-parameter large language model

#81

(creator of gpt index / llamaindex here https://github.com/jerryjliu/gpt_index ) Funny that we had just rebranded our tool from GPT Index to LlamaIndex about a week ago to avoid potential trademark issues with OpenAI, and turns out Meta has similar ideas around LLM+llama puns :). Must mean the name is good though! Also very excited to try plugging in the LLaMa model into LlamaIndex, will report the results.

I look forward to your comparison results.

BTW, I have been heavily experimenting with both your LlamaIndex and also LangChain which you use in LlamaIndex. I am also writing a new book centered around both projects. Great stuff!!

Re: LLaMA: A foundational, 65B-parameter large language model

#82
post #76
post #34

Earlier quoted context omitted.

I don't know what my bias is supposed to be but I called them dorks affectionately for one. The other replies are literally arguing that they are very extremely supervised where as I am speculating they are just eager to share their work for the right reasons and the eye of Sauron has yet to turn upon them. Inside knowledge I never claimed. Anything else I can help you with today? :)

> I called them dorks affectionately Never in my life have I seen that word used affectionately

I have seen that many times, as well as seeing people use it about themselves.

Re: LLaMA: A foundational, 65B-parameter large language model

#83

Unfortunately non-commercial and only available to academics upon request.

darn, guess we'll have to wait until next week when some other surprise organization releases something extremely surprising and useful in this space

Re: LLaMA: A foundational, 65B-parameter large language model

#84
post #17

Earlier quoted context omitted.

How did Microsoft sour developers with Copilot? I know dozens of people that pay for it (including myself) and I feel like it is widely regarded as a "no brainer" for the price that it's offered at. Please help me understand!

Presumably, because they trained Copilot on billions of lines of, often licensed, code (without permission), that Copilot has a tendency to regurgitate verbatim, without said license.

For a specific example some variation of "fast inverse square root" will usually get you the exact GPL licensed code from Quake III, comments included.

Re: LLaMA: A foundational, 65B-parameter large language model

#85
post #47
post #37

Earlier quoted context omitted.

> It seems more likely that a bunch of dorks are running around unsupervised trying their best to make lemonade with whatever budgets they are given Probably not, Zuck is announcing it.

Today we're releasing a new state-of-the-art AI large language model called LLaMA designed to help researchers advance their work. LLMs have shown a lot of promise in generating text, having conversations, summarizing written material, and more complicated tasks like solving math theorems or predicting protein structures. Meta is committed to this open model of research and we'll make our new model available to the A…

It is to please the shareholders. Now shareholder can know that Meta can compete with GPT3.

Re: LLaMA: A foundational, 65B-parameter large language model

#86

Quick notes from first glance at paper https://research.facebook.com/publications/llama-open-and-ef... : * All variants were trained on 1T - 1.4T tokens; which is a good compared to their sizes based on the Chinchilla-metric. Code is 4.5% of the training data (similar to others). [Table 2] * They note the GPU hours as 82,432 (7B model) to 1,022,362 (65B model). [Table 15] GPU hour rates will vary, but let's give a ra…

do they do instruction fine-tuning

Re: LLaMA: A foundational, 65B-parameter large language model

#87

Wow. That's nuts. An open-source model as competitive as 540b Palm? Sign me in. Wish they made it easy to access it, though. Not sure if I would be able to put my hands on it solely as an interested individual.

There's nothing even close to open source here.

Re: LLaMA: A foundational, 65B-parameter large language model

#88
post #10

Earlier quoted context omitted.

> We release all our models to the research community. This is yet more evidence for the "AI isn't a competitive advantage" thesis. State-of-the-Art is a public resource, so competing with AI offers no "moat".

In terms of medieval warfare, what Facebook is doing here looks like filling the moat with rocks and dirt. OpenAI is worth billions, Microsoft is spending billions to retrofit most of their big offerings with AI, Google is doubtless also spending billions to integrate AI in their products. Their moat in all cases is “a blob of X billion weights trained on Y trillion tokens”. Facebook here is spending mere _millions_…

Commoditize your complement.

https://gwern.net/complement

Re: LLaMA: A foundational, 65B-parameter large language model

#89
post #84

Earlier quoted context omitted.

Presumably, because they trained Copilot on billions of lines of, often licensed, code (without permission), that Copilot has a tendency to regurgitate verbatim, without said license.

For a specific example some variation of "fast inverse square root" will usually get you the exact GPL licensed code from Quake III, comments included.

[dead]

Re: LLaMA: A foundational, 65B-parameter large language model

#90
post #11

Earlier quoted context omitted.

I don't see such grand stratagems as being a likely explanation. It seems more likely that a bunch of dorks are running around unsupervised trying their best to make lemonade with whatever budgets they are given as nobody can realistically manage AI researches. Or at least, it seems that much like in political governance, events are suddenly outpacing the ability of corporate to react. In the case of OpenAI it is a "…

I think there is definitely a component of competition here. Facebook has no real way to monetize this (eg they won’t release an api a la OpenAI and they don’t own a search engine). Since they can’t monetize it… why not provide a bit of kindling to lower the barrier for everyone else to compete with your competitors. This strategy is called “commodize your complement”. If Facebook makes it easier to develop a google…

> Facebook has no real way to monetize this (eg they won’t release an api a la OpenAI and they don’t own a search engine). Since they can’t monetize it… why not provide a bit of kindling to lower the barrier for everyone else to compete with your competitors. This strategy is called “commodize your complement”.

Your analysis is good, but that isn't what "commoditize your complement" means.

Strictly speaking, a search engine isn't a complement for FBs revenue streams. Relatively little of FBs revenue can be attributed to search engine traffic leading to FBs walled garden where they can show the user ads.

Complements are generally a required product or service that enables your core, but that isn't revenue generating for you. Examples for FB are data centers (so they participate in the Open Compute Project[0]), and mobile operating systems (which Google already has made a commodity, for their own reasons, with Android).

What FB is doing here is commoditizing their competitors' core offering (or rather, a rather promising future one). That's just the tactic though, there are several strategies this can enable, from undermining the barriers to entry into their competitors' search markets, to actually fragmenting that market by encouraging a diversity of specialized chat interfaces over one big chat model. You can see hints of both in this announcement.

Final note: FB is also protecting itself from relying on a competitor as a supplier should chat become a preferred user interface for the content on social networks, which it hasn't, but if it ever did this would count as "commoditizing their complement", though I would actually expect FB to switch to a mostly proprietary approach in that circumstance (so not much openness on having LLMs operate on social graphs and the like), just keeping the foundation they rely on, and which undermines and prevents gatekeeping by their advertising competitors, open.

[0] https://www.opencompute.org/

Post reply on HN