The MTEB benchmark is dead • Hacker News Board

Re: The MTEB benchmark is dead

#11

Post by RevEng » Tue, Dec 24, 2024, 9:23 PM UTC

cuuupid wrote: post #5

It has been for a while, we ended up building our own test set to evaluate embedding models on our domain. What we realized after doing this is that MTEB has always been a poor indicator, as embedding model performance varies wildly in-domain compared to out-of-domain. You'll get decent performance (lets say 70%) with most models, but eeking out gains over that is domain-dependent more than it is model-dependent. Per…

Unfortunately it requires commercial licensing. I spoke with them a while ago about pricing and it was awfully expensive for being just one part of a larger product. We have been trying other common open source models and the results have been comparable when using them for retrieval on our domain specific data.

Re: The MTEB benchmark is dead

#12

Post by upghost » Tue, Dec 24, 2024, 9:45 PM UTC

light_hue_1 wrote: post #9

upghost wrote: post #7

Earlier quoted context omitted.

I don't know how this is possible with LLM tests. The closed source models will get access to at least the questions when sending the questions over the fence via API. This gives closed source models an enormous advantage over open-source models. The FrontierMath dataset has this same problem[1]. It's a shame because creating these benchmarks is time consuming and expensive. I don't know of a way to fix this except p…

It's possible. I'm not worried about cheaters. We just need to lay out clear rules. You cannot look at the inputs or outputs in any way. You cannot log them. You cannot record them for future use. Either manually or in an automated way. If someone cheats, they will be found out. Their contribution won't stand the test of time, no one will replicate those results with their method. And their performance on datasets th…

I would like to agree with you, but I doubt the honor system will work here. We are talking about companies that have blatantly trampled (or are willing to risk a judicial confrontation about trampling) copyright. It would be unreasonable to assume they would not engage in the same behavior about benchmarks and test sets, especially with the amount of money on the line for the winners.

Re: The MTEB benchmark is dead

#13

Post by greatgib » Wed, Dec 25, 2024, 9:45 AM UTC

0xab wrote: post #2

Datasets need to stop shipping with any training sets at all! And they should forbid anyone from using the test set to update the parameters of any model through their license. We did this with ObjectNet ( https://objectnet.dev/ ) years ago. It's only a test set, no training set provided at all. Back then it was very controversial and we were given a hard time for it initially. Now it's more accepted. Time to make th…

I understand the idea but I don't think that it is beneficial in the end.

Access to the dataset is needed to understand why we get a given result. First from a transparency point of view to check if results make sense and why one model is favored compared to another one.

But also, it is needed to understand why a model will perform badly on some aspect to be able to determine how to improve the model.