Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How big are our embeddings now and why?

vickiboykis.com

Post reply on HN
Print view View original

11–15 of 15 posts

Re: How big are our embeddings now and why?

#11

Post by clickety_clack »

teepo wrote: post #10
minimaxir wrote: post #8

Earlier quoted context omitted.

It's likely because the definition of "similar" varies, and it doesn't necessarily mean semantic similarity. Depending on how the embedding model was trained, just texts with a similar format/syntax are indeed "similar" on that axis. The absolute value of cosine similarity isn't critical (just the order when comparing multiple candidates), but if you finetune an embeddings model for a specific domain, the model will…

Thanks - that helped it click a bit more. If the relative ordering is correct it doesn't matter they look so compressed.

That’s not necessarily true. If the embedding model hasn’t been trained on data you care about, then similarity might be dominated by features you don’t care about. Maybe you want documents that reference pizza toppings, but the embedding similarity might actually be dominated by the tone, word complexity, and the use of em dashes as compared to your prompt. That means the relative ordering might not turn out the way you want.

Re: How big are our embeddings now and why?

#12

Post by numlocked »

indeed30 wrote: post #4
numlocked wrote: post #2

I don’t quite understand. The article says things like: “With the constant upward pressure on embedding sizes not limited by having to train models in-house, it’s not clear where we’ll slow down: Qwen-3, along with many others is already at 4096” But aren’t embedding models separate from the LLMs? The size of attention heads in LLMs etc isn’t inherently connected to how a lab might train and release an embedding mode…

I wouldn’t call the embedding layer "separate" from the LLM. It’s learned jointly with the rest of the network, and its dimensionality is one of the most fundamental architectural choices. You’re right though that, in principle, you can pick an embedding size independent of other hyperparameters like number of layers or heads, so I see where you're coming from. However the embedding dimension sets the rank of the tok…

I hear you, but the article is talking specifically about "embeddings as a product" -- not the embeddings that are within an LLM architecture. It starts:

> As a quick review, embeddings are compressed numerical representations of a variety of features (text, images, audio) that we can use for machine learning tasks like search, recommendations, RAG, and classification.

Current standalone embedding models are not intrinsically connected to SotA LLM architectures (e.g. the Qwen reference) -- right? The article seems to mix the two ideas together.

Re: How big are our embeddings now and why?

#13

Post by lmeyerov »

We have been on a fork for louie.ai:

* Small data - talk to your PDF on-the-fly etc: Getting bigger & faster via cloud APIs

* Big data - for RAG: Getting smaller, bc we don't want to pay crazy fees for vector DB hosting, and doable bc easier to get higher-quality small embeddings that do that

Re: How big are our embeddings now and why?

#14

Post by fredophile »

minimaxir wrote: post #6

It's the same Jevons paradox reason as why LLMs are so big despite massive diminishing returns. If we can output 4096Ds, why not use all the Ds? Like LLMs, the bottleneck is still training data and the training regimen, but there's still a demand for smaller embedding models due to both storage and compute concerns. EmbeddingGemma ( https://huggingface.co/google/embeddinggemma-300m ), released just yesterday, beats t…

I'm not an expert on LLMs but my guess would be that this is a result of the curse of dimensionality. As a general rule more dimensions != more better.
Post reply on HN