41–50 of 217 posts
Is there something like oobabooga to easily run this in a click-and-run way? Where I can load up a model, a text, and ask it questions?
What is the use case for an 8k token embedding? My (somewhat limited) experience with long context models is they aren't great for RAG. I get the impression they are optimized for something else, like writing 8k+ tokens rather than synthesizing responses. Isn't the normal way of using embedding to find relevant text snippets for a RAG prompt? Where is it better to have coarser retrieval?
What is the use case for an 8k token embedding? My (somewhat limited) experience with long context models is they aren't great for RAG. I get the impression they are optimized for something else, like writing 8k+ tokens rather than synthesizing responses. Isn't the normal way of using embedding to find relevant text snippets for a RAG prompt? Where is it better to have coarser retrieval?
Think of it like skipping the square root step in Euclidean distance. Perfectly valid as long as you don’t want a distance so much as a way to compare distances. And doing so skips the most computationally expensive operation.
What is the use case for an 8k token embedding? My (somewhat limited) experience with long context models is they aren't great for RAG. I get the impression they are optimized for something else, like writing 8k+ tokens rather than synthesizing responses. Isn't the normal way of using embedding to find relevant text snippets for a RAG prompt? Where is it better to have coarser retrieval?
Is this what you mean by RAG? https://www.promptingguide.ai/techniques/rag ?
Earlier quoted context omitted.
"text-embedding-ada-002" isn't GPT3, it's a different kind of model. Embedding models and Large Language Models aren't the same thing.
LLMs and embedding models are certainly different, but it's a useful benchmark to calibrate expectations. OpenAI released text-embedding-ada-002 a year ago, and they describe the ada model as[1] "the original GPT-3 base model [...] capable of very simple tasks, usually the fastest model in the GPT-3 series". It's fair to expect GPT3-level results - not GPT 3.5 and certainly not open-source tiny GPT4 as some might thi…
Earlier quoted context omitted.
"text-embedding-ada-002" isn't GPT3, it's a different kind of model. Embedding models and Large Language Models aren't the same thing.
LLMs and embedding models are certainly different, but it's a useful benchmark to calibrate expectations. OpenAI released text-embedding-ada-002 a year ago, and they describe the ada model as[1] "the original GPT-3 base model [...] capable of very simple tasks, usually the fastest model in the GPT-3 series". It's fair to expect GPT3-level results - not GPT 3.5 and certainly not open-source tiny GPT4 as some might thi…
"text-ada-001" is LLM in the GPT3 family, described as "Capable of very simple tasks, usually the fastest model in the GPT-3 series, and lowest cost"
"text-embedding-ada-002" is entirely different - that page describes it as "Our second generation embedding model, text-embedding-ada-002 is a designed to replace the previous 16 first-generation embedding models at a fraction of the cost."
What is the use case for an 8k token embedding? My (somewhat limited) experience with long context models is they aren't great for RAG. I get the impression they are optimized for something else, like writing 8k+ tokens rather than synthesizing responses. Isn't the normal way of using embedding to find relevant text snippets for a RAG prompt? Where is it better to have coarser retrieval?
You could get a facsimile to a summary for a full article or short story. Reducing an 8k token article to a summary using a completions model would cost far more. So if you need to search through collections of contracts, scientific papers, movie scripts, etc. for recommendations/clustering then bigger input sizes can do that in one shot. Think of it like skipping the square root step in Euclidean distance. Perfectly…
I'd much rather know what paragraph to look in than what 25 pages to look in
Earlier quoted context omitted.
You could get a facsimile to a summary for a full article or short story. Reducing an 8k token article to a summary using a completions model would cost far more. So if you need to search through collections of contracts, scientific papers, movie scripts, etc. for recommendations/clustering then bigger input sizes can do that in one shot. Think of it like skipping the square root step in Euclidean distance. Perfectly…
I think I'm missing something: like, yeah, it's vector search for bigger text chunks. But arguably vector search with bigger text chunks is _definitively_ worse -- this isn't doing summarization, just turning about 25 pages of text to 1024 floats, which you then can use cosine similarity to measure the semantic similarity to other text I'd much rather know what paragraph to look in than what 25 pages to look in
What is the use case for an 8k token embedding? My (somewhat limited) experience with long context models is they aren't great for RAG. I get the impression they are optimized for something else, like writing 8k+ tokens rather than synthesizing responses. Isn't the normal way of using embedding to find relevant text snippets for a RAG prompt? Where is it better to have coarser retrieval?
Is this what you mean by RAG? https://www.promptingguide.ai/techniques/rag ?
Some relevant stats from the link: 8192 token input sequence length 768 embedding dimensions 0.27GB model (with 0.07GB model also available) Tokeniser: BertTokenizer [1], 30528 token vocab [2] Is an 8K sequence length directly comparable to text-embedding-ada-002 if the vocabulary is much smaller? I seem to remember its tokeniser has a larger vocabulary. [1] https://huggingface.co/jinaai/jina-embeddings-v2-base-en/bl…