Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta Superintelligence Labs' first paper is about RAG

paddedinputs.substack.com

Post reply on HN
Print view View original

61–70 of 283 posts

Re: Meta Superintelligence Labs' first paper is about RAG

#61

Post by BobbyTables2 »

godelski wrote: post #40
bobxmax wrote: post #28

Earlier quoted context omitted.

I thought Alex Wang was a very curious choice. There are so many foundational AI labs with interesting CEOs... I get that Wang is remarkable in his own right, but he basically just built MTurk and timed the bubble. Doesn't really scream CEO of AGI to me.

A lot of people also don't know that many of the well known papers are just variations on small time papers with a fuck ton more compute thrown at the problem. Probably the strongest feature that correlates to successful researcher is compute. Many have taken this to claim that the GPU poor can't contribute but that ignores so many other valid explanations... and we wonder why innovation has slowed... It's also weird…

It’s funny.

I learnt the hard way that communications/image/signal processing research basically doesn’t care about Computer Architecture at the nuts and bolts level of compiler optimization and implementation.

When they encounter a problem whose normal solution requires excessive amounts of computation, they reduce complexity algorithmically using mathematical techniques, and quantify the effects.

They don’t quibble about a 10x speed up, they reduce the “big O()” complexity. They could care less whether it was implemented in interpreted Python or hand-optimized assembly code.

On one hand, I know there’s a lot of talent in AI today. But throwing hardware at the problem is the dumbest way forward.

WiFI adapters would be wheeled luggage if we had the same mentality during their development.

Re: Meta Superintelligence Labs' first paper is about RAG

#62

Post by hamasho »

godelski wrote: post #15

It's kinda funny, Meta has long had some of the best in the field, but left them untapped. I really think if they just took a step back and stop being so metric focused and let their people freely explore then they'd be winning the AI race. But with this new team, I feel like meta mostly hired the people who are really good at gaming the system. The people that care more about the money than the research. A bit of th…

My theory is that as more people compete, the top candidates become those who are best at gaming the system rather than actually being the best. Someone has probably studied this. My only evidence is job applications for GAFAM and Tinder tho.

Re: Meta Superintelligence Labs' first paper is about RAG

#63

Post by intelkishan »

puttycat wrote: post #42
liampulles wrote: post #11

Earlier quoted context omitted.

If you take the embedding for king, subtract the embedding for male, add the embedding for female, and lookup the closest embedding you get queen. The fact that dot product addition can encode the concept of royalty and gender (among all other sorts) is kind of magic to me.

This was actually shown to not really work in practice.

I have seen this particular work example to work. You don't get the exact match but the closest one is indeed Queen.

Re: Meta Superintelligence Labs' first paper is about RAG

#64

Post by godelski »

hamasho wrote: post #62
godelski wrote: post #15

It's kinda funny, Meta has long had some of the best in the field, but left them untapped. I really think if they just took a step back and stop being so metric focused and let their people freely explore then they'd be winning the AI race. But with this new team, I feel like meta mostly hired the people who are really good at gaming the system. The people that care more about the money than the research. A bit of th…

My theory is that as more people compete, the top candidates become those who are best at gaming the system rather than actually being the best. Someone has probably studied this. My only evidence is job applications for GAFAM and Tinder tho.

  > Someone has probably studied this
There's even a name for it

https://en.wikipedia.org/wiki/Goodhart%27s_law

Re: Meta Superintelligence Labs' first paper is about RAG

#65

Post by godelski »

BobbyTables2 wrote: post #61
godelski wrote: post #40

Earlier quoted context omitted.

A lot of people also don't know that many of the well known papers are just variations on small time papers with a fuck ton more compute thrown at the problem. Probably the strongest feature that correlates to successful researcher is compute. Many have taken this to claim that the GPU poor can't contribute but that ignores so many other valid explanations... and we wonder why innovation has slowed... It's also weird…

It’s funny. I learnt the hard way that communications/image/signal processing research basically doesn’t care about Computer Architecture at the nuts and bolts level of compiler optimization and implementation. When they encounter a problem whose normal solution requires excessive amounts of computation, they reduce complexity algorithmically using mathematical techniques, and quantify the effects. They don’t quibble…

  > They don’t quibble about a 10x speed up, they reduce the “big O()” complexity. They could care less whether it was implemented in interpreted Python or hand-optimized assembly code.
I can at least say that's not all of us. But you're probably right that this is dominating. I find it so weird since everyone stresses empirics yet also seems to not care about them. It took me my entire PhD to figure out what was really going on. I've written too many long winded rants on this site though

Re: Meta Superintelligence Labs' first paper is about RAG

#66

Post by xvector »

incompatible wrote: post #33
xvector wrote: post #13

Working in big tech it's pretty wild to see how integral AI has become to our work internally, vs the public perception of it. People are NOT prepared.

I've heard of one study that said AI slows developers down, even when they think it's helping. https://www.infoworld.com/article/4061078/the-productivity-p...

AI may slow coding a bit but dramatically reduces cognitive load.

The real value of AI isn't in helping coding. It's in having a human-like intelligence to automate processes. I can't get into details but my team is doing things that I couldn't dream of three years ago.

Re: Meta Superintelligence Labs' first paper is about RAG

#67

Post by julienreszka »

godelski wrote: post #64
hamasho wrote: post #62

Earlier quoted context omitted.

My theory is that as more people compete, the top candidates become those who are best at gaming the system rather than actually being the best. Someone has probably studied this. My only evidence is job applications for GAFAM and Tinder tho.

> Someone has probably studied this There's even a name for it https://en.wikipedia.org/wiki/Goodhart%27s_law

It’s a false law tho. Collapses under scrutiny

Re: Meta Superintelligence Labs' first paper is about RAG

#70

Post by shwaj »

BobbyTables2 wrote: post #61
godelski wrote: post #40

Earlier quoted context omitted.

A lot of people also don't know that many of the well known papers are just variations on small time papers with a fuck ton more compute thrown at the problem. Probably the strongest feature that correlates to successful researcher is compute. Many have taken this to claim that the GPU poor can't contribute but that ignores so many other valid explanations... and we wonder why innovation has slowed... It's also weird…

It’s funny. I learnt the hard way that communications/image/signal processing research basically doesn’t care about Computer Architecture at the nuts and bolts level of compiler optimization and implementation. When they encounter a problem whose normal solution requires excessive amounts of computation, they reduce complexity algorithmically using mathematical techniques, and quantify the effects. They don’t quibble…

At some point it becomes difficult to improve the O() complexity. How do you do better that the O(n-squared) of the Transformer, with acceptable tradeoffs? Many big brains in all the big labs are very aware of the importance of algorithmic advances. There is no low hanging fruit, but they're doing their best.

Then in parallel to that looking at compiler optimizations, and other higher-level algorithmic innovations such as Flash Attention (a classic at this point) which had a drastic impact on performance due to cache awareness, without changing the O() complexity.

Post reply on HN