Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jina AI launches open-source 8k text embedding

jina.ai

Post reply on HN
Print view View original

71–80 of 217 posts

Re: Jina AI launches open-source 8k text embedding

#71

Post by SOLAR_FIELDS »

jerpint wrote: post #67
udev4096 wrote: post #62

Earlier quoted context omitted.

Is it tho? It's not really open source if they don't give us the information regarding training datasets

It definitely is open source even if they don’t disclose all details behind the training

The very definition of what constitutes open source is being called into question in these kinds of discussions about AI. Without the training details and the weights being made fully open it’s hard to really call something truly open, even if it happens to meet some arbitrary definition of “open source”.

A good definition of “truly open” is whether the exact same results can be reproduced by someone with no extra information from only what has been made available. If that is not possible, because the reproduction methodology is closed (a common reason, like in this case) then what has been made available is not truly open.

We can sit here and technically argue whether or not the subject matter violated some arbitrary “open source” definition but it still doesn’t change the fact that it’s not truly open in spirit

Re: Jina AI launches open-source 8k text embedding

#72

Post by simonw »

Grimburger wrote: post #70
simonw wrote: post #68

Earlier quoted context omitted.

> You could just sum it up for us all rather than do a divert to your blog? Why? Have links gone out of fashion? I even linked directly to the relevant section rather than linking to the top of the page. The paper that coined the term used the hyphen, though I think I prefer it without: https://arxiv.org/abs/2005.11401

> Have links gone out of fashion? Yes. You wrote far more words than needed to answer the comment, I did it for you instead.

One of the reasons I write so much stuff is so I can provide links to things I've written to answer relevant questions.

Re: Jina AI launches open-source 8k text embedding

#73

Post by simonw »

marinhero wrote: post #58

How well do LLMS like this work with a non-English language? Or are these open source models limited to English?

Quite a few of the top ranked models on this leaderboard are multilingual: https://huggingface.co/spaces/mteb/leaderboard

https://huggingface.co/BAAI/bge-large-en-v1.5 FlagEmbedding for example describes itself as covering Chinese and English.

Re: Jina AI launches open-source 8k text embedding

#74

Post by selcuka »

jerpint wrote: post #67
udev4096 wrote: post #62

Earlier quoted context omitted.

Is it tho? It's not really open source if they don't give us the information regarding training datasets

It definitely is open source even if they don’t disclose all details behind the training

How do you define "source", then?

By this logic any freely downloadable executable software (a.k.a. freeware) is also open source, even though they don't disclose all details on how to build it.

Re: Jina AI launches open-source 8k text embedding

#75

Post by mogwire »

selcuka wrote: post #74
jerpint wrote: post #67

Earlier quoted context omitted.

It definitely is open source even if they don’t disclose all details behind the training

How do you define "source", then? By this logic any freely downloadable executable software (a.k.a. freeware) is also open source, even though they don't disclose all details on how to build it.

Source would be the way the data is produced so that you can replicate it yourself and make changes.

If I hand you a beer for free that’s freeware. If I hand you the recipe and instructions to brew the beer that is open source.

We muddy the waters too much lately and call “free” to use things “open source”.

Re: Jina AI launches open-source 8k text embedding

#78

Post by e1g »

simonw wrote: post #66
e1g wrote: post #53

Earlier quoted context omitted.

OpenAI doesn't say directly what text-embedding-ada-002 is, but in the release blog post they show that performance is comparable to davinci/curie, which places it firmly in the universe of GPT3. I understand it's not a straight line comparison, but to me it's still a useful mental heuristic about what to expect. [1] https://openai.com/blog/new-and-improved-embedding-model (see "Model improvements")

You mean this table here? text-embedding-ada-002 53.3 text-search-davinci-*-001 52.8 text-search-curie-*-001 50.9 text-search-babbage-*-001 50.4 text-search-ada-*-001 49.0 That's not comparing it to the davinci/curie/babbage GPT3 models, it's comparing to the "search-text-*" family. Those were introduced in https://openai.com/blog/introducing-text-and-code-embeddings as the first public release of embeddings models f…

How interesting. I assumed that a consistent codename such as Ada/Davinci refers to the lineage/DNA of the OpenAI model from which a distinct product was created. But I can see how these codenames could be "just" a revision label of A/B/C/D (Ada/Babbage/Curie/Davinci), similar to "Pro/Max/Ultra". If true, a product named "M2 Ultra" could have nothing to do with another product called "Watch Ultra".

Re: Jina AI launches open-source 8k text embedding

#79

Post by lofties »

pknerd wrote: post #54

Pardon my ignorance in advance but could it be used to "chat" with PDFs and websites? I am looking for OpenAI alternatives as I am in learning phase

No. “Chatting with PDFs” is (mostly) taking a users chat message, retrieve relevant content via e.g embedding search, then feed that into an LLM with a prompt that’s something along the lines of “given this information, can you answer this question”.

This tool helps with embedding part.

I’ve built a bunch of ”chat with your PDFs” bots, do reach out if you have any questions me at brian.jp.

Re: Jina AI launches open-source 8k text embedding

#80

Post by HPMOR »

space_fountain wrote: post #30
LoganDark wrote: post #28

Earlier quoted context omitted.

> Is an 8K sequence length directly comparable to text-embedding-ada-002 if the vocabulary is much smaller? I seem to remember its tokeniser has a larger vocabulary. Words that aren't in the vocabulary can still be represented by multiple tokens. Some models can input and output valid UTF-8 at the byte level (rather than needing a unique token for each codepoint). For example RWKV-World.

A large vocabulary means less tokens are needed to represent the same information

*fewer

Less is used for qualitative data like “I love him less”. Whereas fewer is used for countable things like “I need fewer tokens.”

Post reply on HN