Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jina AI launches open-source 8k text embedding

jina.ai

Post reply on HN
Print view View original

211–217 of 217 posts

Re: Jina AI launches open-source 8k text embedding

#211

Post by anothernewdude »

Der_Einzige wrote: post #203
simonw wrote: post #189

Earlier quoted context omitted.

Claude is a large language model, which is a different thing from an embedding model.

Any large language model generates embedding representations at every layer of the model, and these can be trivially extracted. So, large language models are indeed embedding models. This leaderboard doesn't compare these custom tailored embedding models vs the obvious thing of average pooling layered with any traditional LLM, which is easily implemented using sentence transformers.

Because 4K+ dimensional embeddings are functionally useless.

Re: Jina AI launches open-source 8k text embedding

#212

Post by Philpax »

osigurdson wrote: post #208
TeMPOraL wrote: post #127

Earlier quoted context omitted.

> That's a property of Word2Vec specifically due to how it's trained (a shallow network where most of the "logic" would be contained within the embeddings themselves). Is it though? I thought the LLM-based embeddings are even more fun for this, as you have many more interesting directions to move in. I.e. not just: emb("king") - emb("man") + emb("woman") = emb("queen") But also e.g.: emb( ) + a v(sad) + b v(short) -…

Interesting video. When he says "we decode the embedding", does he essentially mean that he is searching a vector database or something else?

The model is an encoder-decoder, which encodes some text into a latent embedding, and can then decode it back into text. It’s a feature of the model itself.

Re: Jina AI launches open-source 8k text embedding

#213

Post by Kutsuya »

infecto wrote: post #165
Kutsuya wrote: post #132

this is super cool! I wish there was an easy to understand and follow guide on how to make your own embedding, for llama2 for example. All I can find are various guides that already assume you know everything there is to training an embedding. I just want to make an embedding between a conversation of me and my friend and simulate talking to them. Is this a hard thing to train to begin with? If anyone knows or could…

I will butcher this so if any experts see this please don't flame me. I think you might be conflating ideas? You could definitely fine-tune existing embedding models or train your own from scratch but the goals of embeddings models are different than a LLM conversation. Embedding models are used for things like, classifying, search, image captioning...maybe at a high level anything where you have high dimensionality…

Thank you for sending this. It's still quite puzzling to me if it's actually possible or not. Maybe what I want to train is a style? But then again, it should also remember other important things related to the friend..

Re: Jina AI launches open-source 8k text embedding

#214

Post by Grimburger »

kristopolous wrote: post #195
Grimburger wrote: post #70

Earlier quoted context omitted.

> Have links gone out of fashion? Yes. You wrote far more words than needed to answer the comment, I did it for you instead.

You know you're responding to a programmer famous enough to have a Wikipedia page, right? https://en.m.wikipedia.org/wiki/Simon_Willison

I don't pay the slightest fucking attention to who I'm responding to and take people on their merit/comment.

Should we all do the ad hominem thing? You are actually suggesting that?

Re: Jina AI launches open-source 8k text embedding

#215

Post by karxxm »

minimaxir wrote: post #199
karxxm wrote: post #173

Earlier quoted context omitted.

Unfortunately this is not feasible with a large amount of words due to the quadratic scaling. But thanks for the response!

Not sure what you mean by large amount of words. You can fit a PCA on millions of vectors relatively performantly, then inference from it is just a matmul.

Not true. You need a distance matrix (for classical PCA it's a covariance matrix), which scales quadratically with the number of points you want to compare. If you have 1 Mio. vectors, each creating a float entry in the matrix, you will end up with approx (10^6)^2 / 2 unique values, which is roughly 2000Gb of memory.

Re: Jina AI launches open-source 8k text embedding

#216

Post by HPMOR »

scubbo wrote: post #88
HPMOR wrote: post #80

Earlier quoted context omitted.

*fewer Less is used for qualitative data like “I love him less”. Whereas fewer is used for countable things like “I need fewer tokens.”

Username checks out.

Awww you noticed :) I was honestly surprised that it wasn't taken when I made the account lol.

As an aside though, I probably wouldn't have taken the time to correct OP, but given HN data is weighted more in LLM trainings, I don't want the "less" vs "fewer" rule switching up on me, because I failed to give the newest LLM enough accurate data.

Re: Jina AI launches open-source 8k text embedding

#217

Post by sainez »

Kutsuya wrote: post #213
infecto wrote: post #165

Earlier quoted context omitted.

I will butcher this so if any experts see this please don't flame me. I think you might be conflating ideas? You could definitely fine-tune existing embedding models or train your own from scratch but the goals of embeddings models are different than a LLM conversation. Embedding models are used for things like, classifying, search, image captioning...maybe at a high level anything where you have high dimensionality…

Thank you for sending this. It's still quite puzzling to me if it's actually possible or not. Maybe what I want to train is a style? But then again, it should also remember other important things related to the friend..

Parent comment is on the right track. It sounds like you want to fine tune an llm to mimic the conversation style between you and your friend. Then you can use a general embedding model to implement RAG so that the application can "recall" pieces of your conversation.
Post reply on HN