Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cohere Launches Embed 4

cohere.com

Post reply on HN
Print view View original

11–20 of 51 posts

Re: Cohere Launches Embed 4

#11

Post by throwup238 »

simonw wrote: post #5

I have huge respect for Cohere and this embedding model looks like it could be best-in-class, but I find it hard to commit to a proprietary embedding model that's only available via an API when there are such good open weight models available. I really like the approach Nomic take: their most recent models are available via their API or as open weights for non-commercial use only (unless you buy a license). They late…

In my experience, a non-finetunable closed source API is a complete nonstarter for a large fraction of possible use cases, especially the higher value ones like law and engineering. Most of these embedding models are trained too much on colloquial use of language on the internet that has little overlap with how terms of art are used, and without the ability to fine tune it to a specific use case, they're only really useful for generic use cases and even then they can be limited.

Re: Cohere Launches Embed 4

#12

Post by moojacob »

esafak wrote: post #6
moojacob wrote: post #4

Seems to under-perform voyage-3-large on the same benchmark. At the same time, I'm unsure how useful benchmarks are for embeddings.

Why? How do you pick an embedding model without benchmarks?

The comment by SparkyMcUnicorn worded it better than I did.

You’re right, there’s no other way to compare embeddings than a benchmark.

Just that what the benchmark used by Voyage and Cohere tracks might not be relevant to your own needs.

Re: Cohere Launches Embed 4

#13

Post by CharlieDigital »

SparkyMcUnicorn wrote: post #8
moojacob wrote: post #4

Seems to under-perform voyage-3-large on the same benchmark. At the same time, I'm unsure how useful benchmarks are for embeddings.

I had the same thought, although voyage is 32k vs 128k for cohere 4. Anecdotal evidence points to benchmarks correlating with result quality for data I've dealt with. I haven't spent a lot of time comparing results between models, because we were happy with the results after trying a few and tuning some settings. Unless my dataset lines up really well with a benchmark's dataset, creating my own benchmark is probably…

Are people using 32k embeddings and no longer chunking?

It feels like embedding content that large -- especially in dense texts -- will lead to loss of fidelity/signal in the output vector.

Re: Cohere Launches Embed 4

#15

Post by mahjongmen »

simonw wrote: post #5

I have huge respect for Cohere and this embedding model looks like it could be best-in-class, but I find it hard to commit to a proprietary embedding model that's only available via an API when there are such good open weight models available. I really like the approach Nomic take: their most recent models are available via their API or as open weights for non-commercial use only (unless you buy a license). They late…

Hey Simon,

Elliott here from Cohere.

We benchmarked against Nomic's models on our consortium of datasets ranging from text-only, image-only, and mixed modalities. Without publishing additional benchmarks, I am confident in saying that our model is more performant.

At Cohere, for our embed models, we have not deprecated any of our embedding models since we started (I know because I've been there that long) and if we were to start doing so, I would take into account the worry of ensuring our users have a way of accessing our models.

One aspect here that isn't factored is also efficiency. Yes there might be strong open weight models but if you're punching at the 7bn+ weight class your serving requirements are vastly different from a throughput efficiency perspective (also your query-inference speed).

All food for thought. That being said, if for your use-case, Nomic Embed Vision 1.5 is better than Embed-v4.0, happy to hop on a call to discuss where the differential may be.

Re: Cohere Launches Embed 4

#16

Post by mahjongmen »

throwup238 wrote: post #11
simonw wrote: post #5

I have huge respect for Cohere and this embedding model looks like it could be best-in-class, but I find it hard to commit to a proprietary embedding model that's only available via an API when there are such good open weight models available. I really like the approach Nomic take: their most recent models are available via their API or as open weights for non-commercial use only (unless you buy a license). They late…

In my experience, a non-finetunable closed source API is a complete nonstarter for a large fraction of possible use cases, especially the higher value ones like law and engineering. Most of these embedding models are trained too much on colloquial use of language on the internet that has little overlap with how terms of art are used, and without the ability to fine tune it to a specific use case, they're only really…

Hey!

Since we focus on Enterprise use-cases, we made sure to include training data from domains like you mentioned above. While in very specific use-cases finetuning may be helpful, but we also do offer that as a customization service (just not available via SaaS)

Re: Cohere Launches Embed 4

#17

Post by simonw »

mahjongmen wrote: post #15
simonw wrote: post #5

I have huge respect for Cohere and this embedding model looks like it could be best-in-class, but I find it hard to commit to a proprietary embedding model that's only available via an API when there are such good open weight models available. I really like the approach Nomic take: their most recent models are available via their API or as open weights for non-commercial use only (unless you buy a license). They late…

Hey Simon, Elliott here from Cohere. We benchmarked against Nomic's models on our consortium of datasets ranging from text-only, image-only, and mixed modalities. Without publishing additional benchmarks, I am confident in saying that our model is more performant. At Cohere, for our embed models, we have not deprecated any of our embedding models since we started (I know because I've been there that long) and if we w…

I don't doubt the new Cohere model is better - but one of the features I value most from an embedding model is having an escape hatch, so I can continue to calculate vectors using that same model far into the future if something happens to the hosting provider.

This matters for embedding models because I'm presumably building up a database of many millions of vectors for later similarity comparisons - so I need to know I'll be able to embed an arbitrary string in the future in order for that investment to still make sense.

Size doesn't matter much to me, I don't even need to be able to run that model, it's more about having an insurance policy for my own peace of mind.

(Even a covenant that says "in the event that Cohere goes out of business this model will be made available under license X" would address this itch for me.)

Re: Cohere Launches Embed 4

#18

Post by mahjongmen »

moojacob wrote: post #4

Seems to under-perform voyage-3-large on the same benchmark. At the same time, I'm unsure how useful benchmarks are for embeddings.

Which benchmark are you referring to?

Voyage-3-large is a text-only and much larger model than Embed-v4. If you want to unlock multimodality with Voyage-3-large, you'd have to either OCR (really bad results usually) or use a VLM to parse your data into textual descriptions (this works alright, but the cost of using a VLM will jack-up your data-pre-processing costs).

Re: Cohere Launches Embed 4

#19

Post by mahjongmen »

cahaya wrote: post #7

Wondering how this compares to the Gemini (preview) embeddings as they seem to perform significantly better than OpenAI embeddings 3 large. I don't see any MTEB scores so hard to compare.

Hey Cahaya,

While we benchmarked internally, on BEIR, we opted not to report our model onto MTEB for the following reason:

1) MTEB has been gamed - if you look at this model (https://huggingface.co/voyageai/voyage-3-m-exp) on the MTEB leaderboard, its an intermediate checkpoint of Voyage-3-Large where they finetuned it on datasets that represent MTEB datasets.

2) If you look at the recent datasets in MMTEB, you'll find that it has quite a lot of machine translated or "weird" datasets that are quite noisy

In general, for our Search Models, we benchmark on these public academic datasets but we definitely do not try to hillclimb in this direction as we find it has little correlation with real use-cases

Re: Cohere Launches Embed 4

#20

Post by mahjongmen »

simonw wrote: post #17
mahjongmen wrote: post #15

Earlier quoted context omitted.

Hey Simon, Elliott here from Cohere. We benchmarked against Nomic's models on our consortium of datasets ranging from text-only, image-only, and mixed modalities. Without publishing additional benchmarks, I am confident in saying that our model is more performant. At Cohere, for our embed models, we have not deprecated any of our embedding models since we started (I know because I've been there that long) and if we w…

I don't doubt the new Cohere model is better - but one of the features I value most from an embedding model is having an escape hatch, so I can continue to calculate vectors using that same model far into the future if something happens to the hosting provider. This matters for embedding models because I'm presumably building up a database of many millions of vectors for later similarity comparisons - so I need to kn…

I'll start off with, I'm not one of our founders and REALLY wouldn't want to be publicly held accountable for policies or commitments until I've been able to get internal alignment on things I say.

That being said, since I do manage our Search and Retrieval offering, if we were to deprecate any of our embedding models (which is generally the risk of closed-source models), I will make sure that there is an "escape hatch" for users.

Heard on what your concerns are though :)

Post reply on HN