Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta Superintelligence Labs' first paper is about RAG

paddedinputs.substack.com

Post reply on HN
Print view View original

271–280 of 283 posts

Re: Meta Superintelligence Labs' first paper is about RAG

#271

Post by danielmarkbruce »

smokel wrote: post #269
danielmarkbruce wrote: post #246

Earlier quoted context omitted.

There is an implication that passionate weirdos are good by nature. You either add value in the world or you don't. A passionate, strange actor or musician who continues trying to "make it" who isn't good enough to be entertaining is a parasite and/or narcissist. A plumber who is doing the job purely for money is a value add (assuming they aren't ripping people off) - and they are playing the game - the money for wor…

I'm not a plumber, but I think the analogy would require the plumber to be strategical about every job they take on. Within a few months, our plumber would only be plumbing for millionaires, installing golden faucets at extreme price points. I would then stop befriending said plumber.

You have a friend who is a plumber, and he figures a way to serve customers with money at high price points with an honest service, and you kick him to the curb?

Re: Meta Superintelligence Labs' first paper is about RAG

#272

Post by smokel »

danielmarkbruce wrote: post #271
smokel wrote: post #269

Earlier quoted context omitted.

I'm not a plumber, but I think the analogy would require the plumber to be strategical about every job they take on. Within a few months, our plumber would only be plumbing for millionaires, installing golden faucets at extreme price points. I would then stop befriending said plumber.

You have a friend who is a plumber, and he figures a way to serve customers with money at high price points with an honest service, and you kick him to the curb?

Extreme price points, not honest service. It's all fairly hypothetical. I don 't know exactly where you'd like the discussion to go. In reality things are always more complicated. I just think that it is a generally a good idea to call out people on anti-social behavior. When, where and how to exactly do that differs in every situation.

Re: Meta Superintelligence Labs' first paper is about RAG

#273

Post by godelski »

danielmarkbruce wrote: post #270
godelski wrote: post #268

Earlier quoted context omitted.

> you are desperate to sound smart. Because I don't think 1-1+1=7? Whatever you say man

You really don't understand that the illogical sounding results from that website are due to the vectors themselves huh. It has zero to do with the definition of +.

Please, tell me more. I was naively under the impression that normal addition had Abelian group properties[0]. Maybe you can inform me as what the inverse element is. That will get me to change my mind

[0] https://en.wikipedia.org/wiki/Abelian_group

Re: Meta Superintelligence Labs' first paper is about RAG

#274

Post by danielmarkbruce »

godelski wrote: post #273
danielmarkbruce wrote: post #270

Earlier quoted context omitted.

You really don't understand that the illogical sounding results from that website are due to the vectors themselves huh. It has zero to do with the definition of +.

Please, tell me more. I was naively under the impression that normal addition had Abelian group properties[0]. Maybe you can inform me as what the inverse element is. That will get me to change my mind [0] https://en.wikipedia.org/wiki/Abelian_group

You’re lost in abstractions. ‘King’ and ‘queen’ and 'man' etc etc aren’t algebraic symbols, they’re mapped to vectors of real numbers. The model learns those mappings, then we just add and subtract numbers element wise. That’s it. You’re giving a group theory lecture about an operation that’s literally just a[i] + b[i]. The semantics come from training, not from some deep mathematical revelation you think everyone missed.

Re: Meta Superintelligence Labs' first paper is about RAG

#275

Post by schmorptron »

edanm wrote: post #248
schmorptron wrote: post #120

One thing I don't get about the ever-reoccuring RAG discussions and hype men proclaiming "Rag is dead", is that people seem to be talking about wholly different things? My mental model is that what is called RAG can either be: - a predefined document store / document chunk store where every chunk gets a a vector embedding, and a lookup decides what gets pulled into context as to not have to pull whole classes of docu…

> My mental model is that what is called RAG can either be: RAG is confusing, because if you look at the words making up the acronym RAG, it seems like it could be either of the things you mentioned. But it originally referred to a specific technique of embeddings + vector search - this was the way it was used in the ML article that defined the term, and this is the way most people in the industry actually use the te…

Okay yeah that makes sense, thanks!

Re: Meta Superintelligence Labs' first paper is about RAG

#276

Post by godelski »

danielmarkbruce wrote: post #274
godelski wrote: post #273

Earlier quoted context omitted.

Please, tell me more. I was naively under the impression that normal addition had Abelian group properties[0]. Maybe you can inform me as what the inverse element is. That will get me to change my mind [0] https://en.wikipedia.org/wiki/Abelian_group

You’re lost in abstractions. ‘King’ and ‘queen’ and 'man' etc etc aren’t algebraic symbols, they’re mapped to vectors of real numbers. The model learns those mappings, then we just add and subtract numbers element wise. That’s it. You’re giving a group theory lecture about an operation that’s literally just a[i] + b[i]. The semantics come from training, not from some deep mathematical revelation you think everyone mi…

  > they’re mapped to vectors of real numbers
Yes, I'm in agreement here. But you need to tell me how

  a - a + a = b
Use what ever the fuck you want for a. A vector (e.g. [1,2,3]), a number (e.g. 1), an embedding (e.g. [[1,2,3],[4,5,6]]), words (e.g. "man"), I really don't give a damn. You have to tell me why b is a reasonable answer to that equation. You have to tell me how a==b while also a!=b.

Because I expect the usual addition to be

  a - a + a = a
This is the last time I'm going to say this to you.

You're telling me I'm lost in abstraction and I'm telling you is not usual addition because a != b. That's it! That's the whole fucking argument. You literally cannot see the contradiction right in front of you. The only why it is usual addition is if you tell me "man == woman" because that is literally the example from several comments ago. Stop being so smart and just read the damn comment

Re: Meta Superintelligence Labs' first paper is about RAG

#277

Post by ekianjo »

phkahler wrote: post #213
HPsquared wrote: post #162

Earlier quoted context omitted.

This is the right terminology. Model weights are literally compiled binary data; they are the output of an algorithm run on a bunch of source data. That training dataset is the "source" of the model. Training data (or the scripts used to generate it) is human-readable and modifiable, like source code. Binary weights are not.

Binary weights can still be "edited" with additional training.

Binary images can also be edited after they are created.

Re: Meta Superintelligence Labs' first paper is about RAG

#278

Post by ekianjo »

drexlspivey wrote: post #148
ekianjo wrote: post #136

Earlier quoted context omitted.

Open weights models, not open source. And even their weights are under a specific license not as permissive as apache 2.

Does an “open source” model the way you describe it exist or is it a mythical creature?

I believe it does. OLMO.

Re: Meta Superintelligence Labs' first paper is about RAG

#279

Post by danielmarkbruce »

godelski wrote: post #276
danielmarkbruce wrote: post #274

Earlier quoted context omitted.

You’re lost in abstractions. ‘King’ and ‘queen’ and 'man' etc etc aren’t algebraic symbols, they’re mapped to vectors of real numbers. The model learns those mappings, then we just add and subtract numbers element wise. That’s it. You’re giving a group theory lecture about an operation that’s literally just a[i] + b[i]. The semantics come from training, not from some deep mathematical revelation you think everyone mi…

> they’re mapped to vectors of real numbers Yes, I'm in agreement here. But you need to tell me how a - a + a = b Use what ever the fuck you want for a. A vector (e.g. [1,2,3]), a number (e.g. 1), an embedding (e.g. [[1,2,3],[4,5,6]]), words (e.g. "man"), I really don't give a damn. You have to tell me why b is a reasonable answer to that equation. You have to tell me how a==b while also a!=b. Because I expect the us…

a - a + a = b when a and b map to the same vector (or in practice, extremely close together). Your assumptions about invertibility etc don't hold in this world.... embeddings are just a bunch of empirically learned coordinates in a dense space.

So an example: a maps to [1,2,3] and b maps to [1,2,3] . Again in practice b could map to [1,2,3.0001] or something.

To summarize: king, man etc aren't symbols, they get mapped to vectors. + is element wise addition. = is "equal to or very close in multi dimensional space".

Maybe tone down the attitude. You clearly aren't in this field. The properties you have assumed to be true are not. People in AI/ML are using terms and conventions differently than you assume. When someone says "vector addition" they really do mean just element wise addition in practically every case. You are the fool here.

Re: Meta Superintelligence Labs' first paper is about RAG

#280

Post by godelski »

danielmarkbruce wrote: post #279
godelski wrote: post #276

Earlier quoted context omitted.

> they’re mapped to vectors of real numbers Yes, I'm in agreement here. But you need to tell me how a - a + a = b Use what ever the fuck you want for a. A vector (e.g. [1,2,3]), a number (e.g. 1), an embedding (e.g. [[1,2,3],[4,5,6]]), words (e.g. "man"), I really don't give a damn. You have to tell me why b is a reasonable answer to that equation. You have to tell me how a==b while also a!=b. Because I expect the us…

a - a + a = b when a and b map to the same vector (or in practice, extremely close together). Your assumptions about invertibility etc don't hold in this world.... embeddings are just a bunch of empirically learned coordinates in a dense space. So an example: a maps to [1,2,3] and b maps to [1,2,3] . Again in practice b could map to [1,2,3.0001] or something. To summarize: king, man etc aren't symbols, they get mappe…

  man - man + man = woman
  woman - woman + woman = man
  => man = woman

  > Your assumptions about invertibility etc don't hold in this world
Yes? Thats what I've said lol. That's what the above example shows. THAT WAS THE ENTIRE POINT

  > So an example: a maps to [1,2,3] and b maps to [1,2,3] . Again in practice b could map to [1,2,3.0001] or something.

  >>>>>>>>>> Floating point arithmetic is not associative.
I'm glad you finally decided to agree with me. But it would have been a lot faster had you actually read my comments.
Post reply on HN