Can we please get rid of the clickbait titles?
Meta Superintelligence Labs' first paper is about RAG
141–150 of 283 posts
Re: Meta Superintelligence Labs' first paper is about RAG
#142One thing I don't get about the ever-reoccuring RAG discussions and hype men proclaiming "Rag is dead", is that people seem to be talking about wholly different things? My mental model is that what is called RAG can either be: - a predefined document store / document chunk store where every chunk gets a a vector embedding, and a lookup decides what gets pulled into context as to not have to pull whole classes of docu…
Technically, RAG is anything that augments generation with external search. However, it often has a narrower meaning: "uses a vector DB."
Throwing everything into one large context window is often impractical - it takes much more time to process, and many models struggle to find information accurately if too much is going on in the context window ("lost in the middle").
The "classic" RAG still has its place when you want low latency (or you're limited by VRAM) and the results are already good enough.
Re: Meta Superintelligence Labs' first paper is about RAG
#143Earlier quoted context omitted.
My theory is that as more people compete, the top candidates become those who are best at gaming the system rather than actually being the best. Someone has probably studied this. My only evidence is job applications for GAFAM and Tinder tho.
I've spent most of my career working, chatting and hanging out with what might be best described as "passionate weirdos" in various quantitative areas of research. I say "weirdos" because they're people driven by an obsession with a topic, but don't always fit the mold by having the ideal combination of background, credentials and personality to land them on a big tech company research team. The other day I was spend…
This is such a nuanced problem. Like any creative endeavour, the most powerful and significant research is driven by an innate joy of learning, creating, and sharing ideas with others. How far the research can be taken is then shaped by resource constraints. The more money you throw at the researchers, the more results they can get. But there seems to be a diminishing returns kind of effect as individual contributors become less able to produce results independently. The research narrative also gets distorted by who has the most money and influence, and not always for the better (as recent events in Alzheimer's research has shown).
The problem is once people's livelihoods depend on their research output rather than the research process, the whole research process becomes steadily distorted to optimise for being able to reliably produce outputs.
Anyone who has invested a great deal of time and effort into solving a hard problem knows that the 'eureka' moment is not really something that you can force. So people end up spending less time working on problems that would contribute to 'breakthroughs' and more time working on problems that will publish.
Re: Meta Superintelligence Labs' first paper is about RAG
#144Earlier quoted context omitted.
> I certainly didn't judge them because they are just playing the game. Please do judge them for being parasitical. They might seem successful by certain measures, like the amount of money they make, but I for one simply dislike it when people only think about themselves. As a society, we should be more cautious about narcissism and similar behaviors. Also, in the long run, this kind of behaviour makes them an annoyi…
But this is in itself selfish right? You dislike them because they don’t benefit you indirectly by benefiting society at large. The incentive structure is wrong, incentivizing things that benefit society would be the solution not judging those that exist in the current system by pretending altruism is somehow not part of the same game.
I agree that the system itself is dysfunctional, and I understand the argument that individuals are shaped or even constrained by it. However, in this case, we are talking about people who are both exceptionally intelligent and materially secure. I think it's reasonable to expect such individuals to feel some moral responsibility to use their abilities for broader good.
As for whether that expectation is "selfish" on my part, I think that question has been debated for centuries in ethics, and I'm quite comfortable landing on the side that says not all disapproval is self-interest. In my own case, I'm not benefiting much either :)
Re: Meta Superintelligence Labs' first paper is about RAG
#145Earlier quoted context omitted.
My theory is that as more people compete, the top candidates become those who are best at gaming the system rather than actually being the best. Someone has probably studied this. My only evidence is job applications for GAFAM and Tinder tho.
But there is no way to know who is truly the 'best'. The people who position and market themselves to be viewed as the best are the only ones who even have a chance to be viewed as such. So if you're a great researcher but don't project yourself that way, no one will ever know you're a great researcher (except for the other great researchers who aren't really invested in communicating how great you are). The system s…
The problem is that the best research is not a competitive process but a collaborative one. Positioning research output as a race or a competition is already problematic.
Re: Meta Superintelligence Labs' first paper is about RAG
#146One thing I don't get about the ever-reoccuring RAG discussions and hype men proclaiming "Rag is dead", is that people seem to be talking about wholly different things? My mental model is that what is called RAG can either be: - a predefined document store / document chunk store where every chunk gets a a vector embedding, and a lookup decides what gets pulled into context as to not have to pull whole classes of docu…
no one is saying rag is dead, you're never going to put the whole Internet in the context of the model, & the more you put the more expensive it is.
Re: Meta Superintelligence Labs' first paper is about RAG
#147Earlier quoted context omitted.
> I certainly didn't judge them because they are just playing the game. Please do judge them for being parasitical. They might seem successful by certain measures, like the amount of money they make, but I for one simply dislike it when people only think about themselves. As a society, we should be more cautious about narcissism and similar behaviors. Also, in the long run, this kind of behaviour makes them an annoyi…
But this is in itself selfish right? You dislike them because they don’t benefit you indirectly by benefiting society at large. The incentive structure is wrong, incentivizing things that benefit society would be the solution not judging those that exist in the current system by pretending altruism is somehow not part of the same game.
There is a difference between being selfish in the sense that you want others to contribute back to the society that we are all part of, and being selfish in the sense that you want to compete for exclusive rewards.
You can call this difference whatever you want, don't pretend that they are morally or effectively equivalent.
Re: Meta Superintelligence Labs' first paper is about RAG
#148This has nothing to do with superintelligence, it's just the people that were working on the paper prior to the re-org happened to publish after the name change. Though it is notable that contrary to many (on HN and Twitter) that Meta would stop publishing papers and be like other AI labs (e.g. OpenAI). They're continued their rapid pace of releasing papers AND open source models.
Open weights models, not open source. And even their weights are under a specific license not as permissive as apache 2.
Does an “open source” model the way you describe it exist or is it a mythical creature?
Re: Meta Superintelligence Labs' first paper is about RAG
#149Earlier quoted context omitted.
it doesn't break down - see comments about rules above. it was the perfect example to prove yourself wrong.
I disagree with all of those examples, they are misunderstanding what it means for the metric to break down in the context of the law, but alas. "If you run a different race" lol.
could you explain what you think the difference is?
a metric is chosen, people start to game the system by doing things that make the metric improve but the original intent is lost. increasingly specific rules/laws have to be made up to make the metric appear to work, but it becomes a lost cause as more and more creative ways are found to work around the rules.
Re: Meta Superintelligence Labs' first paper is about RAG
#150Earlier quoted context omitted.
I have seen this particular work example to work. You don't get the exact match but the closest one is indeed Queen.
Yes but it doesn't generalize very well. Even on simple features like gender. If you go look at embeddings you'll find that man and woman are neighbors, just as king and queen are[0]. This is a better explanation for the result as you're just taking very small steps in the latent space. Here, play around[1] mother - parent + man = woman father - parent + woman = man father - parent + man = woman mother - parent + wom…
so addition is not associative?