AI for AWS Documentation • Hacker News Board

Re: AI for AWS Documentation

#131

Post by underlines » Fri, Jul 07, 2023, 9:21 PM UTC

d4rkp4ttern wrote: post #115

underlines wrote: post #58

RAG is very difficult to do right. I am experimenting with various RAG projects from [1]. The main problems are: - Chunking can interfer with context boundaries - Content vectors can differ vastly from question vectors, for this you have to use hypothetical embeddings (they generate artificial questions and store them) - Instead of saving just one embedding per text-chuck you should store various (text chunk, hypothe…

That’s an interesting list (awesome-marketing-data science). Curious what is specific to marketing in that list, or maybe it’s just an inaccurate name.

historical repo name, it's really not that anymore, besides a very old list of marketing stuff that i rarely update. I should rename the repo, but I hesitate :)

Re: AI for AWS Documentation

#132

Post by underlines » Fri, Jul 07, 2023, 9:30 PM UTC

avereveard wrote: post #79

underlines wrote: post #58

RAG is very difficult to do right. I am experimenting with various RAG projects from [1]. The main problems are: - Chunking can interfer with context boundaries - Content vectors can differ vastly from question vectors, for this you have to use hypothetical embeddings (they generate artificial questions and store them) - Instead of saving just one embedding per text-chuck you should store various (text chunk, hypothe…

To solve the question embedding issue I prefer another approach, you store document embeddings as normal, then for searching you let LLM hallucinate an answer and use the wrongish answer embedding to do the vector search.

The inverse idea of Hypothetical Embeddings, HyDE [1] "HyDE is an embedding technique that takes queries, generates a hypothetical answer, and then embeds that generated document and uses that as the final example."

BriefGPT [2] is implementing this and it uses the following prompt at ingestion-time:

"Given the user's question, please generate a response that mimics the exact format in which the relevant information would appear within a document, even if the information does not exist. The response should not offer explanations, context, or commentary, but should emulate the precise structure in which the answer would be found in a hypothetical document. Factuality is not important, the priority is the hypothetical structure of the excerpt. Use made-up facts to emulate the structure. For example, if the user question is "who are the authors?", the response should be something like 'Authors: John Smith, Jane Doe, and Bob Jones' The user's question is:"

1 https://python.langchain.com/docs/modules/chains/additional/...

2 https://github.com/e-johnstonn/BriefGPT

Re: AI for AWS Documentation

#133

Post by avereveard » Sun, Jul 09, 2023, 9:47 AM UTC

CSDude wrote: post #108

avereveard wrote: post #78

Earlier quoted context omitted.

It still isn't tho, this will work for testing, maybe, but each lambda will be it's own connection and that will exhaust db resources real fast, you're supposed to have a pooling proxy between lambdas and RDS.

And it's in the docs, RDS Proxy, launched in 2020, before GPT training cut-off https://aws.amazon.com/about-aws/whats-new/2020/06/amazon-rd...

Cut off date shouldn't matter for RAG

Re: AI for AWS Documentation

#134

Post by scarface_74 » Sun, Jul 09, 2023, 4:54 PM UTC

BoorishBears wrote: post #129

scarface_74 wrote: post #125

Earlier quoted context omitted.

That technique didn’t work when I asked it to create a Python script to return all IAM roles with a certain set of policies attached. It still missed using a paginator to handle the list_roles call returning more than 50 roles. Once I pointed it out, it did add pagination support.

I try not to cheat and hint at specific issues (since that relies on prior knowledge) I'd be surprised if even after the last prompt it wouldn't notice that. Saying "Did we miss anything" leaves it open it to re-evaluate both the implementation and the original considerations Edit: There's some non-determinism involved, but GPT-4 caught the pagination from planning stage here: https://chat.openai.com/share/3c356d4f-1…

Funny enough, even in my original question, it put pagination as a consideration. But it still didn’t include it.