Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Contextual Retrieval

anthropic.com

Post reply on HN
Print view View original

61–70 of 73 posts

Re: Contextual Retrieval

#61

Post by _kb »

jiggawatts wrote: post #40
williamcotton wrote: post #36

Earlier quoted context omitted.

Could you explain or link to explanations of all of the acronyms you’ve used in your comment?

It makes me chuckle a bit to see this kind of request in a tech forum, particularly when discussing advanced LLM-related topics. This is akin to a HN comment asking someone to search the Internet for something on their behalf, while discussing search engine algorithms!

A lot of people here (myself included) work across different specialisations and are here to learn from discussion that is intentionally unfamiliar.

Re: Contextual Retrieval

#62

Post by jiggawatts »

_kb wrote: post #61
jiggawatts wrote: post #40

Earlier quoted context omitted.

It makes me chuckle a bit to see this kind of request in a tech forum, particularly when discussing advanced LLM-related topics. This is akin to a HN comment asking someone to search the Internet for something on their behalf, while discussing search engine algorithms!

A lot of people here (myself included) work across different specialisations and are here to learn from discussion that is intentionally unfamiliar.

Yes, but ChatGPT knows these things! Just ask it to expand the acronyms.

This is the new “can you Google that for me?”

Re: Contextual Retrieval

#63

Post by _kb »

jiggawatts wrote: post #62
_kb wrote: post #61

Earlier quoted context omitted.

A lot of people here (myself included) work across different specialisations and are here to learn from discussion that is intentionally unfamiliar.

Yes, but ChatGPT knows these things! Just ask it to expand the acronyms. This is the new “can you Google that for me?”

يمكن لـ ChatGPT أيضًا الترجمة من العربية إلى الإنجليزية، ولكن سيكون من المزعج استخدامه للمحادثة في هذا السياق

Re: Contextual Retrieval

#64

Post by jiggawatts »

_kb wrote: post #63
jiggawatts wrote: post #62

Earlier quoted context omitted.

Yes, but ChatGPT knows these things! Just ask it to expand the acronyms. This is the new “can you Google that for me?”

يمكن لـ ChatGPT أيضًا الترجمة من العربية إلى الإنجليزية، ولكن سيكون من المزعج استخدامه للمحادثة في هذا السياق

Annyira lusta vagyok, hogy nem akarok néhány gombot megnyomni, ezért kérlek, írj nekem egy oldal szöveget.

Re: Contextual Retrieval

#65

Post by layoric »

The statement about just throwing 200k tokens to get best answer for smaller datasets goes against my experience. I commonly find as my prompt gets larger, the less consistent the output becomes, and the poorer following instructions becomes. Does anyone else experience this or a well known way to avoid this? It seems to happen at much less than even 25k tokens.

Re: Contextual Retrieval

#66

Post by whereismyacc »

msp26 wrote: post #35

> If your knowledge base is smaller than 200,000 tokens (about 500 pages of material) I would prefer that anthropic just release their tokeniser so we don't have to make guesses.

shouldn't this be possible to reverse-engineer since they stream the responses token-by-token?

Re: Contextual Retrieval

#67

Post by whereismyacc »

vendiddy wrote: post #23

I don't know anything about AI but I've always wished I could just upload a bunch of documents/books and the AI would perform some basic keyword searches to figure out what is relevant, then auto include that in the prompt.

i've noticed that copilot seems to do this pretty well, i ask about a function and it correctly looks up the relevant lines of code

Re: Contextual Retrieval

#68

Post by santiagobasulto »

williamcotton wrote: post #36
underlines wrote: post #27

We build a corporate RAG for a government entity. What I've learned so far by applying an experimental A/B testing approach to RAG using RAGAS metrics: - Hybrid Retrieval (semantic + vector) and then LLM based Reranking made no significant change using synthetic eva-questions - HyDE decreased answer quality and retrieval quality severly when measured with RAGAS using synthetic eval-questions (we still have to do a RA…

Could you explain or link to explanations of all of the acronyms you’ve used in your comment?

These are all "techniques" on top of the foundations of RAG. It's similar to "Chain of Thought" in prompt engineering. You have an underlying technology, and then come up with techniques/frameworks on top. What MVC was for Web dev +15 years ago.

RAPTOR for example is a technique that groups and clusters documents together, summarizes them, and creates embeddings defining a sort of a Tree. Paper: https://arxiv.org/html/2401.18059v1

Agentic RAG is creating an agent that can decide to augment "conversations" (or other LLM tools) with RAG searches and analyze its relevance. Pretty useful, but hard to implement right.

You can google the others, they're all more or less these "techniques" to improve an old-fashioned RAG search.

Re: Contextual Retrieval

#69

Post by sebastiennight »

torginus wrote: post #25
skeptrune wrote: post #3

I'm not a fan of this technique. I agree the scenario they lay out is a common problem, but the proposed solution feels odd. Vector embeddings have bag-of-words compression properties and can over-index on the first newline separated text block to the extent that certain indices in the resulting vector end up much closer to 0 than they otherwise would. With quantization, they can eventually become 0 and cause you to…

Sorry random question - do vector dbs work across models? I'd guess no, since embeddings are models specific afaik, but that means that a vector db would lock you into using a single LLM and even within that, a single version, like Claude-3.5 Sonnet, and you couldn't move to 3.5 Haiku, Opus etc., never mind ChatGPT or Llama without reindexing.

The sibling comments seem to be correct in their technical explanations, but miss the meaning I'm getting from your question.

My understanding is you want to know "are vector DBs compatible with specific LLMs, or are we stuck with a specific LLM if we want to do RAG once we've adopted a specific vector store?"

And the answer to that is that the LLM never sees the vectors from your DB. Your LLM only sees what you submit as context (ie the "system" and "user" prompts in chat-based models).

The way RAG works is:

1 - end-user submits a query

2 - this query is embedded (with the same model that was used to compile the vector store) and compared (in the vector store) with existing content, to retrieve relevant chunks of data

3 - and then this data (in the form of text segments) is passed to the LLM along with the initial query.

So, in a sense you're "locked in" in the sense that you need to use the same embedding model for storage and for retrieval. But you can definitely swap out the LLM for any other LLM without reindexing.

An easy way to try this behavior out as a layperson is to use AnythingLLM which is an open-source desktop client, that allows you to embed your own documents and use RAG locally with open-weight models or swap out any of the LLM APIs.

Re: Contextual Retrieval

#70

Post by sebastiennight »

ValentinA23 wrote: post #47

Interesting. One problem I'm facing is using RAG to retrieve applicable rules instead of knowledge (chunks): only rules that may apply to the context should be injected into the context. I haven't done any experiment, but one approach that I think could work would be to train small classifiers to determine whether a specific rule could apply. The main LLM would be tasked with determining whether the rule indeed appli…

If the in-game context is properly described in the query, I believe that the same traditional vector search used in RAG would match your case.

Example query, with some help from LLama 3.1 8B:

    As the dark elven horde closes in on his position, Grimgold Ironfist finds himself in a desperate predicament. His sturdy bearded face is set with determination, but his worn leather apron and mismatched socks are a far cry from the battle-hardened armor he once donned as a proud member of the Dwarven Militia. Now, his tunic is stained with ale and oil from a recent session at the local tavern, and his boots are scuffed from countless miles of adventuring. His health bar, once a proud 100%, now teeters on 35% due to a nasty encounter with a giant spider earlier that day. In his inventory, Grimgold has: a rusty iron pickaxe (degraded), a waterskin (half-full), a chunk of stale bread (half-eaten), and a small pouch containing 17 gold pieces. His trusty hammer, "Mithrilcrusher", lies forgotten in the nearby underbrush, having been left behind in his haste to flee the elven army. With no time to lose, Grimgold spots a lone taxi cab rattling down the road - its golden horse emblem a beacon of hope in this desperate hour. He sprints towards it, hoping against hope that he can somehow sweet-talk the driver into taking him on, despite his...ahem...' checkered past' with the Taxi Guild.
Example rule that would be fetched from the vector store (because there is a vector proximity caused by the character name/attributes and by the mentions of taxis and the Taxi Guild.

   The Taxi Guild has imposed a strict penalty upon Grimgold: whenever he attempts to hail a cab, he is summarily ejected from the vehicle. The Guild’s decree, inscribed on a parchment of shame, reads:
    “Grimgold Ironfist, bearded dwarf of ill repute, henceforth shall not be granted passage in any taxi operated by our members until he has formally apologized to Thorgrim Stonebeard, Director of the Golden Horse Cab Company. Failure to comply with this edict shall result in perpetual exclusion from our services.”
Post reply on HN