Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jina AI launches open-source 8k text embedding

jina.ai

Post reply on HN
Print view View original

31–40 of 217 posts

Re: Jina AI launches open-source 8k text embedding

#31

Post by simonw »

moralestapia wrote: post #27

Ada is one of the (if not the) worst model offered by OpenAI, though ...

You're thinking of the old "ada" GPT-3 model - the one that was a companion to "davinci" and "babbage".

I believe "text-embedding-ada-002" is entirely unrelated to those old GPT-3 models. It's a recent embedding model (released in December 2022 - https://openai.com/blog/new-and-improved-embedding-model ) which OpenAI claim is their best current best available embedding model.

I understand your confusion: OpenAI are notoriously bad at naming things!

Re: Jina AI launches open-source 8k text embedding

#32

Post by moralestapia »

simonw wrote: post #31
moralestapia wrote: post #27

Ada is one of the (if not the) worst model offered by OpenAI, though ...

You're thinking of the old "ada" GPT-3 model - the one that was a companion to "davinci" and "babbage". I believe "text-embedding-ada-002" is entirely unrelated to those old GPT-3 models. It's a recent embedding model (released in December 2022 - https://openai.com/blog/new-and-improved-embedding-model ) which OpenAI claim is their best current best available embedding model. I understand your confusion: OpenAI are n…

Oh, thanks for clarifying!

Edit: looking at the press release, the improvement over old Ada is ... marginal? And Ada-01 is/was a poor performing model, tbh. I guess I'll have to run some tests, but at first sight it doesn't seem that wow-ey.

Re: Jina AI launches open-source 8k text embedding

#34

Post by minimaxir »

jncraton wrote: post #14

This is great to see. It looks like the size of the embedding vector is half the size of text-embedding-ada-002 (768 vs 1536) while providing competitive performance. This will save space in databases and make lookups somewhat faster. For those unaware, if 512 tokens of context is sufficient for your use case, there are already many options that outperform text-embedding-ada-002 on common benchmarks: https://huggingf…

The 768D-sized embeddings compared to OpenAI's 1536D embeddings are actually a feature outside of index size.

In my experience, OpenAI's embeddings are overspecified and do very poorly with cosine similarity out of the box as they match syntax more than semantic meaning (which is important as that's the metric for RAG). Ideally you'd want cosine similarity in the range of [-1, 1] on a variety of data but in my experience the results are [0.6, 0.8].

Re: Jina AI launches open-source 8k text embedding

#37

Post by [deleted] »

RossBencina wrote: post #9

Some relevant stats from the link: 8192 token input sequence length 768 embedding dimensions 0.27GB model (with 0.07GB model also available) Tokeniser: BertTokenizer [1], 30528 token vocab [2] Is an 8K sequence length directly comparable to text-embedding-ada-002 if the vocabulary is much smaller? I seem to remember its tokeniser has a larger vocabulary. [1] https://huggingface.co/jinaai/jina-embeddings-v2-base-en/bl…

[deleted]

Re: Jina AI launches open-source 8k text embedding

#38

Post by dazzaji »

simonw wrote: post #21

I just shipped a new llm-embed-jina plugin for my LLM tool which provides access to these new Jina models: https://github.com/simonw/llm-embed-jina Here's how to try it out. First, install LLM. Use pip or pipx or brew: brew install llm Next install the new plugin: llm install llm-embed-jina You can confirm the new models are now available to LLM by running: llm embed-models You should see a list that includes "jina-e…

Excellent! And you were just saying how risky it is to rely long-term on OpenAI text embeddings in your post on the topic. The timing for this open source option worked out nicely.

Re: Jina AI launches open-source 8k text embedding

#39

Post by osigurdson »

BoorishBears wrote: post #15
andrewstuart wrote: post #5

Anyone got links to examples of text embedding?

Easiest example is taking three words: Universe, University, College. - University and Universe are similar alphabetically. - University and College are similar in meaning. Take embeddings for those three words and `University` will be near `College`, while `Universe` will be further away, because embeddings capture meaning: University College Universe _ With old school search you'd need to handle the special case of…

Another interesting point is that math can be performed on embedding vectors: emb("king") - emb("man") + emb("woman") = emb("queen").

Re: Jina AI launches open-source 8k text embedding

#40

Post by e1g »

simonw wrote: post #24
e1g wrote: post #16

Their OpenAI benchmark is GPT3 (text-embedding-ada-002), not GPT4.

"text-embedding-ada-002" isn't GPT3, it's a different kind of model. Embedding models and Large Language Models aren't the same thing.

LLMs and embedding models are certainly different, but it's a useful benchmark to calibrate expectations. OpenAI released text-embedding-ada-002 a year ago, and they describe the ada model as[1] "the original GPT-3 base model [...] capable of very simple tasks, usually the fastest model in the GPT-3 series".

It's fair to expect GPT3-level results - not GPT 3.5 and certainly not open-source tiny GPT4 as some might think when they read "rivaling OpenAI".

[1] https://platform.openai.com/docs/models/whisper

Post reply on HN