Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cohere Launches Embed 4

cohere.com

Post reply on HN
Print view View original

31–40 of 51 posts

Re: Cohere Launches Embed 4

#31

Post by mahjongmen »

wrs wrote: post #21
mahjongmen wrote: post #20

Earlier quoted context omitted.

I'll start off with, I'm not one of our founders and REALLY wouldn't want to be publicly held accountable for policies or commitments until I've been able to get internal alignment on things I say. That being said, since I do manage our Search and Retrieval offering, if we were to deprecate any of our embedding models (which is generally the risk of closed-source models), I will make sure that there is an "escape hat…

To someone building a long term dataset, I’m not sure what assurances would help. Certainly a personal assurance doesn’t (though you’re kind to offer), and even a corporate statement doesn’t (new owners or C-suite could walk that back anytime). It might take a formal third-party “model escrow” arrangement to be really convincing.

Hey All,

Thanks for engaging! Apologies for the delay but HN seems to have throttled my account from posting so I'm answering as fast as I can (or they will let me).

You're right in the sense that I could wake up tomorrow and Cohere could lay me off, fire me, or I could quit! All of these are possible statements, but the reason I don't want to publicly commit particularly on our policy on Open Sourcing our models if our business is a going concern or if we deprecate our models is as follows:

1) Cohere is not a going concern 2) I haven't thought about deprecating any of our embedding models because of the reason that simonw stated!

I wouldn't say I'm a Cohere employee playing PR - I'm responsible for all the search and embedding models and products at Cohere and I care deeply on how our users perceive, understand and user our models/products. I'm actually really excited that there is so much engagement this time around (a far cry from 2021 - when I started).

For reference in terms of policies: For our SaaS API, I wrote our model deprecation policy (https://docs.cohere.com/docs/deprecations) and had only deprecated our Rerank-v2.0 Models largely because they were stateless

Again - happy for all the engagement. Heard on the things we can improve on!

Re: Cohere Launches Embed 4

#32

Post by mahjongmen »

serjester wrote: post #25
mahjongmen wrote: post #18

Earlier quoted context omitted.

Which benchmark are you referring to? Voyage-3-large is a text-only and much larger model than Embed-v4. If you want to unlock multimodality with Voyage-3-large, you'd have to either OCR (really bad results usually) or use a VLM to parse your data into textual descriptions (this works alright, but the cost of using a VLM will jack-up your data-pre-processing costs).

I think anyone that cares enough about embedding performance to use niche models is probably parsing their PDF's into some sort of textual format. Otherwise you need orient your all your pipelines to handle images which adds significant complexity (hybrid search, reranking, LLM calls, etc - all way harder with images). Not to mention an image is optimistically 50 KB vs the same page represented as markdown is maybe 2…

Hey Serjester Email me at elliott@cohere.ai, let's arrange time to chat. We did head to head evals with Voyage Large / Voyage Multimodal and I can share them with you if you are serious about moving your embeddings over. We tested configurations of top open-source, closed-source, multi-vector and single-dense embedding models but I can only choose so many to put on a graph and I'm not in the business of giving Voyage free advertising haha. I agree with you that there is some complexity on multi-modal reranking w.r.t to inference time speeds as well as data transfer / network latency costs. Happy to talk more :)

Re: Cohere Launches Embed 4

#33

Post by mahjongmen »

lukebuehler wrote: post #3

I just started to look into multi-modal embedding models recently, and I was surprised how few options there are. For example, Google's model only supports 30 text tokens [1]!! This is definitely a welcome addition. Any pointers to similarly powerful embedding models? I'm looking specifically for text and images? I wish there'd be also one that could do audio and video, but I don't think that exists. [1] https://clou…

Hey Luke, Our model does exceptionally well on text and images, and in particular, when text and images are mixed together. An example of where this works well would be in E-commerce where you may have a product title, description, and multiple images of the product. When combining that into a single payload using our inputs parameter we find that our model responds really well to adding more images (i.e. retrieval quality moves up as you add 1,2,3....N images). As you pointed out with Google's multimodal model, most jointly trained multimodal embedding models will suffer in the text modality. Amazon used to have a multimodal embedding model, which also took in a very small text payload. We're thinking about Audio / Video as well but nothing for Q2 at least....

Re: Cohere Launches Embed 4

#34

Post by andriym »

mahjongmen wrote: post #15
simonw wrote: post #5

I have huge respect for Cohere and this embedding model looks like it could be best-in-class, but I find it hard to commit to a proprietary embedding model that's only available via an API when there are such good open weight models available. I really like the approach Nomic take: their most recent models are available via their API or as open weights for non-commercial use only (unless you buy a license). They late…

Hey Simon, Elliott here from Cohere. We benchmarked against Nomic's models on our consortium of datasets ranging from text-only, image-only, and mixed modalities. Without publishing additional benchmarks, I am confident in saying that our model is more performant. At Cohere, for our embed models, we have not deprecated any of our embedding models since we started (I know because I've been there that long) and if we w…

Hey Elliot,

Andriy, co-founder at Nomic here! Congrats on Embed v4 - the more embeddings the merrier!

Embed v1.5 is a 1.5 year old model!

You should check out our latest comparable open-weights, multimodal embedding model that's designed for text, PDFs and images! I can't directly say anything about relative performance to Embed v4 as you guys didn't publish evals on the Vidore-V2 open benchmark!

https://www.nomic.ai/blog/posts/nomic-embed-multimodal

Re: Cohere Launches Embed 4

#35

Post by [deleted] »

mahjongmen wrote: post #15
simonw wrote: post #5

I have huge respect for Cohere and this embedding model looks like it could be best-in-class, but I find it hard to commit to a proprietary embedding model that's only available via an API when there are such good open weight models available. I really like the approach Nomic take: their most recent models are available via their API or as open weights for non-commercial use only (unless you buy a license). They late…

Hey Simon, Elliott here from Cohere. We benchmarked against Nomic's models on our consortium of datasets ranging from text-only, image-only, and mixed modalities. Without publishing additional benchmarks, I am confident in saying that our model is more performant. At Cohere, for our embed models, we have not deprecated any of our embedding models since we started (I know because I've been there that long) and if we w…

[deleted]

Re: Cohere Launches Embed 4

#36

Post by pencildiver »

I'm a huge fan of Cohere. We were highlighted in the launch post and use their V3 text embeddings in production: https://www.searchagora.com/

We're switching to the V4 to store unified embeddings of our products. From the early tests we ran, this should help with edge case relevancy (i.e. when a product's image and text mismatch, thus creating a greater need for multi-modal embeddings) and improve our search speed by ~100ms.

Re: Cohere Launches Embed 4

#38

Post by BrandiATMuhkuh »

This is really great. I'll use it asap. I'm working with enterprise clients in the AEC space. Having a model that actually understands documents with messy data (drawings, floor plans, books, norms, ...) will be great.

The current situation of chunking and transforming is such a messy situation.

Re: Cohere Launches Embed 4

#39

Post by mahjongmen »

andriym wrote: post #34
mahjongmen wrote: post #15

Earlier quoted context omitted.

Hey Simon, Elliott here from Cohere. We benchmarked against Nomic's models on our consortium of datasets ranging from text-only, image-only, and mixed modalities. Without publishing additional benchmarks, I am confident in saying that our model is more performant. At Cohere, for our embed models, we have not deprecated any of our embedding models since we started (I know because I've been there that long) and if we w…

Hey Elliot, Andriy, co-founder at Nomic here! Congrats on Embed v4 - the more embeddings the merrier! Embed v1.5 is a 1.5 year old model! You should check out our latest comparable open-weights, multimodal embedding model that's designed for text, PDFs and images! I can't directly say anything about relative performance to Embed v4 as you guys didn't publish evals on the Vidore-V2 open benchmark! https://www.nomic.ai…

Hey Andriy!

We actually did internally run benchmarks against your models since they are open-weights - however, when looking at the license on the 3bn multimodal model (https://huggingface.co/nomic-ai/nomic-embed-multimodal-3b/bl...) we're not permitted to include the results for the marketing of products/services. Rest assured, we know how our model stacks up against yours :)

In any-case, we didn't publish evals on only Vidore-V2 but we did benchmark on it internally.

Re: Cohere Launches Embed 4

#40

Post by mahjongmen »

BrandiATMuhkuh wrote: post #38

This is really great. I'll use it asap. I'm working with enterprise clients in the AEC space. Having a model that actually understands documents with messy data (drawings, floor plans, books, norms, ...) will be great. The current situation of chunking and transforming is such a messy situation.

That sounds really cool! Would love to better understand your use-case and make sure it works well for you!

Drop me an email at elliott@cohere.ai

Post reply on HN