Vector Database benchmark with 1536/768 dim data
1–10 of 13 posts
Re: Vector Database benchmark with 1536/768 dim data
#2The vendors performance metrics rate their own product the highest marks.
Re: Vector Database benchmark with 1536/768 dim data
#3The vendors performance metrics rate their own product the highest marks.
Previous discussions from last time they posted this: https://news.ycombinator.com/item?id=36856815
If they’re going to rank themselves so much higher than their competition, they might as well call that out up front and explain why the discrepancy is so large.
Re: Vector Database benchmark with 1536/768 dim data
#4The vendors performance metrics rate their own product the highest marks.
It's really hard to benchmark this sort of a thing. There are so many layers of caching and external factors that play into it, from all manner of sources including the operating system load and configuration, disk firmware, hardware configuration, and so forth; and the harder you try to isolate these effects, the farther you get from a realistic benchmark because all the factors that were removed are affecting real world performance in a big way.
This is a big reason why for a long time many large DBMS-providers had clauses in their licenses prohibiting 3rd party benchmarks. You can fairly easily construct a benchmark that makes any given DBMS seem great or awful, and there's no such thing as an objective test.
Re: Vector Database benchmark with 1536/768 dim data
#5The vendors performance metrics rate their own product the highest marks.
Yes, of course vendors have bias. But IMHO, if a benchmark is reproducible and the use cases can match users' needs, then we can say it can somehow help decision making.
Re: Vector Database benchmark with 1536/768 dim data
#6Beyond the other grains of salt, it seems awfully inorganic that the same user advertises this database 6 times as their only submission history.
Re: Vector Database benchmark with 1536/768 dim data
#7The vendors performance metrics rate their own product the highest marks.
It's really hard to benchmark this sort of a thing. There are so many layers of caching and external factors that play into it, from all manner of sources including the operating system load and configuration, disk firmware, hardware configuration, and so forth; and the harder you try to isolate these effects, the farther you get from a realistic benchmark because all the factors that were removed are affecting real…
Fully agree with this idea. All tricks can be a real world strategy and it is impossible for anyone to claiming that they have an absolute fair benchmark.
So the only way we can do to approaching it is to provide more real-world-like cases and forget all tricks vendors might play inside their systems.
Also, people will concern the representative of the cases benchmarks provide. So we plan to make this benchmark more like a framework to support customized cases in the next step.
Re: Vector Database benchmark with 1536/768 dim data
#8Beyond the other grains of salt, it seems awfully inorganic that the same user advertises this database 6 times as their only submission history.
Yeah.. this combined with the fact that this benchmark happens to rank their cloud offering the highest by a wide margin sounds a bit like they are submitting it to market themselves.
Re: Vector Database benchmark with 1536/768 dim data
#9Beyond the other grains of salt, it seems awfully inorganic that the same user advertises this database 6 times as their only submission history.
Generally these, say, too "proactive" moves to artificially gain attention to your own GitHub projects makes me less likely to test it out, so I'd rather stay with the mainstream options.
Re: Vector Database benchmark with 1536/768 dim data
#10Beyond the other grains of salt, it seems awfully inorganic that the same user advertises this database 6 times as their only submission history.
I apologize for the repeated posts. The reason I posted it 6 times is that I aim to announce every release and significant commit. The reason my entire history is centered around this benchmark is because I wanted to introduce my project to the community, potentially with some bias. I began my Hacker News journey at that time and wanted to share what I was working on.