Jina AI launches open-source 8k text embedding • Hacker News Board

Re: Jina AI launches open-source 8k text embedding

#121

Post by otikik » Thu, Oct 26, 2023, 7:54 AM UTC

SOLAR_FIELDS wrote: post #71

jerpint wrote: post #67

Earlier quoted context omitted.

It definitely is open source even if they don’t disclose all details behind the training

The very definition of what constitutes open source is being called into question in these kinds of discussions about AI. Without the training details and the weights being made fully open it’s hard to really call something truly open, even if it happens to meet some arbitrary definition of “open source”. A good definition of “truly open” is whether the exact same results can be reproduced by someone with no extra in…

Well the other day on this very website there were some very opinionated voices stating that Open Source is “exclusively what OSI defines”. I am not on that camp, more like in yours. To me there’s open source and OSI-approved open source. But you will encounter people very set on that other opinion, which I found interesting.

Make no mistake, I am super grateful to OSI for their efforts and most of my code out there uses one of their licenses. I just think they are limited by the circumstances. Some things I consider open are not conforming to their licenses and, like here, some things that conform might not be really open.

Re: Jina AI launches open-source 8k text embedding

#122

Post by 3abiton » Thu, Oct 26, 2023, 7:55 AM UTC

dragonwriter wrote: post #61

andy99 wrote: post #26

What is the use case for an 8k token embedding? My (somewhat limited) experience with long context models is they aren't great for RAG. I get the impression they are optimized for something else, like writing 8k+ tokens rather than synthesizing responses. Isn't the normal way of using embedding to find relevant text snippets for a RAG prompt? Where is it better to have coarser retrieval?

> What is the use case for an 8k token embedding? Calculating embeddings on larger documents than smaller-window embedding models. > My (somewhat limited) experience with long context models is they aren't great for RAG. The only reason they wouldn't be great for RAG is that they aren't great at using information in their context window, which is possible (ISTR that some models have a strong recency bias within the w…

I wonder how the perfomance fair when context size is increased. Intuitively this should be higher, but some quantized models I've tested showed noticeably worst performance.

Re: Jina AI launches open-source 8k text embedding

#123

Post by LASR » Thu, Oct 26, 2023, 7:56 AM UTC

moralestapia wrote: post #32

simonw wrote: post #31

Earlier quoted context omitted.

You're thinking of the old "ada" GPT-3 model - the one that was a companion to "davinci" and "babbage". I believe "text-embedding-ada-002" is entirely unrelated to those old GPT-3 models. It's a recent embedding model (released in December 2022 - https://openai.com/blog/new-and-improved-embedding-model ) which OpenAI claim is their best current best available embedding model. I understand your confusion: OpenAI are n…

Oh, thanks for clarifying! Edit: looking at the press release, the improvement over old Ada is ... marginal? And Ada-01 is/was a poor performing model, tbh. I guess I'll have to run some tests, but at first sight it doesn't seem that wow-ey.

So just to be super clear, this is an embedding model. It generates no text. It’s not outputting words.

Maybe I am assuming incorrectly, but I think the poor performance you are referring to is the old Ada completion model, where the output is text. That was poor indeed.

Re: Jina AI launches open-source 8k text embedding

#124

Post by imranhou » Thu, Oct 26, 2023, 8:00 AM UTC

TeMPOraL wrote: post #107

mogwire wrote: post #75

Earlier quoted context omitted.

Source would be the way the data is produced so that you can replicate it yourself and make changes. If I hand you a beer for free that’s freeware. If I hand you the recipe and instructions to brew the beer that is open source. We muddy the waters too much lately and call “free” to use things “open source”.

> If I hand you a beer for free that’s freeware. If I hand you the recipe and instructions to brew the beer that is open source. Yeah, but what those "open source" models are is like you handing me a bottle of beer, plus the instructions to make the glass bottle . You're open-sourcing something, just not the part that matters. It's not "open source beer", it's "beer in an open-source bottle". In the same fashion, tho…

Perhaps one more thing that is missing in context is that I'm also getting the right to alter that beer by adding anything I like to it and redistributing it, without knowing its true recipe.

Re: Jina AI launches open-source 8k text embedding

#125

Post by jyrkesh » Thu, Oct 26, 2023, 8:02 AM UTC

abriosi wrote: post #115

SOLAR_FIELDS wrote: post #71

Earlier quoted context omitted.

The very definition of what constitutes open source is being called into question in these kinds of discussions about AI. Without the training details and the weights being made fully open it’s hard to really call something truly open, even if it happens to meet some arbitrary definition of “open source”. A good definition of “truly open” is whether the exact same results can be reproduced by someone with no extra in…

Imagine someone giving you a executable binary without the source code and calling it "open source"

I'm actually mostly in your camp here. But it's complicated with AI.

What if someone gave you a binary and the source code, but not a compiler? Maybe not even a language spec?

Or what if they gave you a binary and the source code and a fully documented language spec, and both of 'em all the way down to the compiler? BUT it only runs on special proprietary silicon? Or maybe even the silicon is fully documented, but producing that silicon is effectively out of reach to all but F100 companies?

It's turtles all the way down...

Re: Jina AI launches open-source 8k text embedding

#126

Post by DougBTX » Thu, Oct 26, 2023, 8:02 AM UTC

abriosi wrote: post #115

SOLAR_FIELDS wrote: post #71

Earlier quoted context omitted.

The very definition of what constitutes open source is being called into question in these kinds of discussions about AI. Without the training details and the weights being made fully open it’s hard to really call something truly open, even if it happens to meet some arbitrary definition of “open source”. A good definition of “truly open” is whether the exact same results can be reproduced by someone with no extra in…

Imagine someone giving you a executable binary without the source code and calling it "open source"

You can pass in any command line arguments you like, so it must be open source

Re: Jina AI launches open-source 8k text embedding

#127

Post by TeMPOraL » Thu, Oct 26, 2023, 8:09 AM UTC

minimaxir wrote: post #65

osigurdson wrote: post #39

Earlier quoted context omitted.

Another interesting point is that math can be performed on embedding vectors: emb("king") - emb("man") + emb("woman") = emb("queen").

That's a property of Word2Vec specifically due to how it's trained (a shallow network where most of the "logic" would be contained within the embeddings themselves). Using it for embeddings generated from LLMs or Embedding layers will not give as fun results; in practice the only thing you can do is average or cluster them.

> That's a property of Word2Vec specifically due to how it's trained (a shallow network where most of the "logic" would be contained within the embeddings themselves).

Is it though? I thought the LLM-based embeddings are even more fun for this, as you have many more interesting directions to move in. I.e. not just:

emb("king") - emb("man") + emb("woman") = emb("queen")

But also e.g.:

emb() + av(sad) + bv(short) - c*v(positive) = emb()

Where a, b, c are some constants to tweak, and v(X) is a vector for quality X, which you can get by embedding a bunch of texts expressing the quality X and averaging them out (or doing some other dimensional reduction trickery).

--

[0] - https://www.youtube.com/watch?v=veShHxQYPzo&t=13980s - "The Hidden Life of Embeddings", by Linus Lee from Notion.

Re: Jina AI launches open-source 8k text embedding

#128

Post by imranhou » Thu, Oct 26, 2023, 8:11 AM UTC

rolisz wrote: post #55

simonw wrote: post #48

Earlier quoted context omitted.

I imagine it's more useful for finding related articles and clustering things than for semantic search, which will work much better against smaller chunks - especially if you're implementing Retrieval Augmented Generation.

I think the point is: if you compress 25 pages of text into 1024 floats, you will lose a ton of information, regardless of what the use case is, so you're probably still better of with chunking.

Good point, I wonder how different it is to use a large context here vs having some other model summarize an 8k article into a small paragraph and using embedding from the paragraph instead where such a large context wouldn't be necessary.

Re: Jina AI launches open-source 8k text embedding

#129

Post by m3at » Thu, Oct 26, 2023, 8:23 AM UTC

SOLAR_FIELDS wrote: post #71

jerpint wrote: post #67

Earlier quoted context omitted.

It definitely is open source even if they don’t disclose all details behind the training

The very definition of what constitutes open source is being called into question in these kinds of discussions about AI. Without the training details and the weights being made fully open it’s hard to really call something truly open, even if it happens to meet some arbitrary definition of “open source”. A good definition of “truly open” is whether the exact same results can be reproduced by someone with no extra in…

To take an other example, would you call a game that has its code and all assets (ex. character sprites) freely available open source? Or would the process that was used to create the assets in the first place also be required to be considered open?

The parallel can be made with model weights being static assets delivered in their completed state.

(I favor the full process being released especially for scientific reproducibility, but this is an other point)

Re: Jina AI launches open-source 8k text embedding

#130

Post by magicalhippo » Thu, Oct 26, 2023, 8:27 AM UTC

losteric wrote: post #92

okaram wrote: post #86

Earlier quoted context omitted.

Notice you are creating your own arbitrary definition of 'truly open', which IMHO corresponds more with 'reproducible'. We already have a definition of open source. I don't see any reason to change it.

The inference runtime software is open, the weights are an opaque binary. Publishing the training data, hyperparameters, process, etc - that would make the whole thing "open source".

The quake engine is still open source even though it doesn't come with the quake game assets, no?

It seems unreasonable to require the training data just to be called open source, given it has similar copyright challenges as game assets.

Of course, this wouldn't make the model reproducible. But that's different from open source.