I couldn't immediately see in their graphs/tables any comparison against simple lexical/statistical based context compression, such as candidate selection of chunks using TF-IDF, word overlap etc. For most of us in the industry we need to find these quick wins that give us equivalent performance to sending huge amount of information to the LLM, while compressing by 10x.
Meta Superintelligence Labs' first paper is about RAG
131–140 of 283 posts
Re: Meta Superintelligence Labs' first paper is about RAG
#132Earlier quoted context omitted.
> So what is your argument, that it doesn't apply everywhere therefore it applies nowhere? I never said that. Someone said the law collapses, someone asked for a link, I gave an example to prove it does break down in some cases at least, but many cases once you think more about it. I never said all cases. If it works sometimes and not others, it's not a law. It's just an observation of something that can happen or no…
it doesn't break down - see comments about rules above. it was the perfect example to prove yourself wrong.
I disagree with all of those examples, they are misunderstanding what it means for the metric to break down in the context of the law, but alas. "If you run a different race" lol.
Re: Meta Superintelligence Labs' first paper is about RAG
#133So this looks essentially like continuous prompting (see prefix tuning) with RL-driven selection of what to present as tokens and what as continuous inputs (embeddings).
Re: Meta Superintelligence Labs' first paper is about RAG
#134Earlier quoted context omitted.
My theory is that as more people compete, the top candidates become those who are best at gaming the system rather than actually being the best. Someone has probably studied this. My only evidence is job applications for GAFAM and Tinder tho.
I've spent most of my career working, chatting and hanging out with what might be best described as "passionate weirdos" in various quantitative areas of research. I say "weirdos" because they're people driven by an obsession with a topic, but don't always fit the mold by having the ideal combination of background, credentials and personality to land them on a big tech company research team. The other day I was spend…
> I certainly didn't judge them because they are just playing the game.
Please do judge them for being parasitical. They might seem successful by certain measures, like the amount of money they make, but I for one simply dislike it when people only think about themselves.
As a society, we should be more cautious about narcissism and similar behaviors. Also, in the long run, this kind of behaviour makes them an annoying person at parties.
Re: Meta Superintelligence Labs' first paper is about RAG
#135this was really weird to read: > But RAG is a very real world, practical topic for something as significant as a new lab’s first paper. I would expect exactly the opposite - that a new lab would put out a few random papers that happen to be in areas their researchers were interested in and already working on, and once people had been working together a while and developed some synergy they would maybe come out with s…
As an academic I would expect the same as you, and no, to my knowledge "first paper" is meaningless, at least in academia. Most people's first paper is some small contribution to what their PhD supervisor is doing at the time, where the student tries their best at writing but it ends up so heavily edited that probably 90% of the final text comes from the supervisor :) So typically first papers don't define or represent a researcher. When you start you just don't have the experience to have a great idea and carry it through to a good paper.
Of course here we are talking about a lab, not an individual person, but still I haven't heard of first papers being considered special in any way, even for labs.
Re: Meta Superintelligence Labs' first paper is about RAG
#136This has nothing to do with superintelligence, it's just the people that were working on the paper prior to the re-org happened to publish after the name change. Though it is notable that contrary to many (on HN and Twitter) that Meta would stop publishing papers and be like other AI labs (e.g. OpenAI). They're continued their rapid pace of releasing papers AND open source models.
Open weights models, not open source. And even their weights are under a specific license not as permissive as apache 2.
Re: Meta Superintelligence Labs' first paper is about RAG
#137Earlier quoted context omitted.
I've spent most of my career working, chatting and hanging out with what might be best described as "passionate weirdos" in various quantitative areas of research. I say "weirdos" because they're people driven by an obsession with a topic, but don't always fit the mold by having the ideal combination of background, credentials and personality to land them on a big tech company research team. The other day I was spend…
> I certainly didn't judge them because they are just playing the game. Please do judge them for being parasitical. They might seem successful by certain measures, like the amount of money they make, but I for one simply dislike it when people only think about themselves. As a society, we should be more cautious about narcissism and similar behaviors. Also, in the long run, this kind of behaviour makes them an annoyi…
[dead]
Re: Meta Superintelligence Labs' first paper is about RAG
#138Earlier quoted context omitted.
It’s a false law tho. Collapses under scrutiny
Sorry, remind me; how many cobras are there in India?
The Zoological Survey of India would like to know but hasn't figured out a good way to do a full census. If you have any ideas they would love to hear them.
Naja naja has Least Concern conservation status, so there isn't much funding in doing a full count, but there are concerns as encroachment both reduces their livable habitat and puts them into more frequent contact with humans and livestock.
Re: Meta Superintelligence Labs' first paper is about RAG
#139Earlier quoted context omitted.
I've spent most of my career working, chatting and hanging out with what might be best described as "passionate weirdos" in various quantitative areas of research. I say "weirdos" because they're people driven by an obsession with a topic, but don't always fit the mold by having the ideal combination of background, credentials and personality to land them on a big tech company research team. The other day I was spend…
> I certainly didn't judge them because they are just playing the game. Please do judge them for being parasitical. They might seem successful by certain measures, like the amount of money they make, but I for one simply dislike it when people only think about themselves. As a society, we should be more cautious about narcissism and similar behaviors. Also, in the long run, this kind of behaviour makes them an annoyi…
But this is in itself selfish right?
You dislike them because they don’t benefit you indirectly by benefiting society at large.
The incentive structure is wrong, incentivizing things that benefit society would be the solution not judging those that exist in the current system by pretending altruism is somehow not part of the same game.
Re: Meta Superintelligence Labs' first paper is about RAG
#140Earlier quoted context omitted.
Sorry, remind me; how many cobras are there in India?
The Zoological Survey of India would like to know but hasn't figured out a good way to do a full census. If you have any ideas they would love to hear them. Naja naja has Least Concern conservation status, so there isn't much funding in doing a full count, but there are concerns as encroachment both reduces their livable habitat and puts them into more frequent contact with humans and livestock.
The comment was a joke.