121–130 of 132 posts
Earlier quoted context omitted.
Would a bigger model be able to beat yours if some effort were put into prompt?
the bigger model would still cost more :) at the same time, i see prompting as being orthogonal to post-training. i'd imagine post-training a smaller model with a better prompt would make it perform even better
Earlier quoted context omitted.
There has been an over-obsession with frontier models and benchmarks. Most of the work will be done by task specific models. You don't put Phds on the factory floor.
If the PhDs don't cost very much more than your equivalent of factory-technicians but still get the job done, why wouldn't you do that, at least in the blunt case before cost control rears up?
Earlier quoted context omitted.
Product search model: https://www.linkedin.com/posts/introducing-ontology-1-ugcPos... Edit: more direct links, sorry: https://onton.com/research/ontology-1 https://onton.com/research/ontology-1-benchmarks
Sorry but what is neurosymbolic model, it's not explained I think
What does "100x cheaper" mean? Costs are 1 / 100? If so, is there a more appropriate term to use?
cheaper in terms of token costs
Basic embeddings give decent-ish results (in the top say 20 chunks). Basic agentic retrieval gives slightly better results so long as the agent part of it doesn't go down the wrong track.
I like the idea of what's discussed in the link, however atm we are on Bedrock KBs and so locked in to a very basic implementation of RAG, because Amazon doesn't have the foresight to make things flexible enough - including making it an absolute pita to use their hybrid search. But, I guess they "work" reliably.
One of our core issues centers around a 1300 page document all about the same overall topic but with minor various for specific procedures/situations. Typical embeddings waters this down so that each chunk really just represents the common theme and therefore lacks a lot of contrast.
But now that luna's (and others) price has been cut, perhaps I'll start experimenting with giving it free rein to explore the data a little in the same way that I do a web search.
One thing that definitely helped was providing a separate index of each section where I had another model summarise the primary unique topics in each section to act as a guide for the agent. I think either we should be chucking the entire doc at a model (400k tokens...so not really ideal at this time) or improving RAG accuracy. For the latter I think even with embeddings, meaning of words and semantic connections are not enough at all - attention is KV so it is 2 dimensional and once I started getting into it I've kind of realised that 2 dimensions aren't really enough to represent the logic that exists between tokens (i.e. sections of documents that refer to a sequence of actions dependent on some logic that references "variables" from another section, i.e. "if x, y has happened then refer to z sequence). There's much deeper meaning to human language than I think basic embeddings covers.
I think it's becoming clear to me that in the same way that embeddings encode the web of semantic meaning of a chunk of text, I need something similar to a hybrid of the author's model + reranker + super-embeddings that encodes as much of the entire meaning of a text as possible and not just semantic.
People are building agentic search one of three ways: 1. Actually good retireval. There’s been a lot of progress on serving the kinds of queries agents tend to serve, from places like Hornet, MoxedBread, LightOn. Particularly in late interaction 2. Smarter harnesses with models/judges validating the result. This is now just seen as the generator/ evaluator pattern. Here’s where people try to just use grep or some oth…
It seems like a lot of the problems I have been running into with RAG on large/complex documents with generally low contrast in the information is not one that has been perfectly solved yet - here I am thinking I'd been a bit behind.
It's just unfortunate that none of the cloud providers are flexible enough to deal with the pace of change. Probably going to have to shove one of those 8b~ models into an instance to use when needed.
It's interesting you mention late interaction (retrieval), I had recently been using ChatGPT as a mirror to throw ideas back at me on this issue and had been musing about how nice it would be to have some sort of hierarchical embeddings that capture a whole chunk, then sentences and then sentence fragments or individual word and it seems that that fits the bill!
There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks like this. I know this is not a novel idea. Claude code does some of this by handing off the "explore" agent work to haiku. I just love seeing that specialized LLMs are being developed.
There are! Chroma has Context1, SID has SID-1, and you'd actually be surprised at how easy it is to post-train your own with pretty good pass@ recall@ ndcg@ etc. There's also Hornet who have shared some interesting talks & blogs lately. I don't know that I'd exclusively use agents for retrieval the way Neon outlines here as well. I think distillation similar to what ZeroEntropy has done for bespoke retrieval & rerank…
I haven't looked into model architecture people are working with for this stuff too deeply yet but I presume the core idea is fine-tuning a lightweight reasoning-enabled LLM specifically using search as a metric for training?
There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks like this. I know this is not a novel idea. Claude code does some of this by handing off the "explore" agent work to haiku. I just love seeing that specialized LLMs are being developed.
IMO, purpose built or “adapted” models are The Next Big Thing. If I’m using a model to write Python code, for instance, I really want the 8B or 27B expert model for exactly that, which would also be runnable locally. I don’t care about the 1.8T model that can answer every query under the sun and that only runs in a remote data center.
I am wondering if models will end up being some sort of evolution of MoE where it has something internally like the model the author refers to that gets surfaced when it needs to search in some way. I guess it makes sense; our own brains have so many distinct task-specific regions.
There is a more serious question in here that's not being answered. How effective is the retrieval in finding buried needles in larger and larger haystacks. And there's a correlary question, how effective could you be in finding paired needles in that haystack where you need to hold a needle to unlock finding another needle.
I've found with traditional embeddings that obviously you're getting an average of the content of the chunk even with the semantic awareness magic. And our (or I guess the) core problem seems to be a lack of enough contrast between chunks with makes one-shot pure embedding based RAG extremely difficult and low quality.