21–30 of 44 posts
This benchmark is pretty ridiculous for the following reasons: 1. Their database is run in asynchronous durability mode. 2. They specifically do the one thing that TPC-C says you shouldn't do, which is get really high throughput on a small dataset. TPC-C enforces that you scale your data-stored with the query throughput. CockroachDB maxes out at ~12.8tpmC/warehouse because its waiting at the legal maximum throughput…
Earlier quoted context omitted.
Because all a benchmark tells you is the result of one use case. It's not an estimate of what your use will result in. It's basically a sales pitch. And sadly, upper management falls for it all the time, often without an evaluation period or even spitballing it for a week with the teams that would be using it. Tons of factors will change the result for a particular use case, regardless of how "normalized" the test tr…
The whole point of benchmarks is that their "use case" correlates roughly with every use case related to the technology being benchmarked. Is the correlation 1:1? No. Is it still relevant when making a decision about what technology to use? Absolutely.
That's the idea of benchmarks, but it's not actually true of real benchmarks, because invariably some factors which can improve a particular benchmarks beaean inverse relation to performance for some other use cases of the technology.
Earlier quoted context omitted.
The whole point of benchmarks is that their "use case" correlates roughly with every use case related to the technology being benchmarked. Is the correlation 1:1? No. Is it still relevant when making a decision about what technology to use? Absolutely.
> The whole point of benchmarks is that their "use case" correlates roughly with every use case related to the technology being benchmarked. That's the idea of benchmarks, but it's not actually true of real benchmarks, because invariably some factors which can improve a particular benchmarks beaean inverse relation to performance for some other use cases of the technology.
Earlier quoted context omitted.
> The whole point of benchmarks is that their "use case" correlates roughly with every use case related to the technology being benchmarked. That's the idea of benchmarks, but it's not actually true of real benchmarks, because invariably some factors which can improve a particular benchmarks beaean inverse relation to performance for some other use cases of the technology.
The problem you're having is that everybody already knows that, and plenty of folks can still extract value out of a benchmark anyway.
I'm not having a problem.
> is that everybody already knows that,
Certainly, some people act like they don't, at least in the context of specific benchmarks.
And some people outright claim the opposite of what you say everyone understands, e.g., by claiming that benchmarks inherently correlate with every possible use of a technology.
> and plenty of folks can still extract value out of a benchmark anyway.
Understanding both the general issue and, ideally, the specific areas of potential concern in relation to particular benchmarks and your intended use is a big part of being able to effectively extract value from a benchmark.and, yes, lots of people do recognize those facts and extract value from benchmarks.
Others don't, and still apply benchmarks in decision-making, but it's less clear that they are extracting value. Confusing the measurement most readily available with the measurement most relevant to need is a common problem (and not just with benchmarks.)
As someone who's spent a lot of time working with TPC-DS [1] and talking to people about it [2], I see a couple areas that could be improved in this benchmark: 1. Total run time is not an appropriate way to summarize the performance across queries, because some queries take 100x longer than others. The appropriate way to summarize this kind of data is to use the geomean [3]. 2. The official TPC-DS queries make heavy…
Curious to see TiDB in there
Earlier quoted context omitted.
The problem you're having is that everybody already knows that, and plenty of folks can still extract value out of a benchmark anyway.
> The problem you're having I'm not having a problem. > is that everybody already knows that, Certainly, some people act like they don't, at least in the context of specific benchmarks. And some people outright claim the opposite of what you say everyone understands, e.g., by claiming that benchmarks inherently correlate with every possible use of a technology. > and plenty of folks can still extract value out of a b…
You're throwing up your hands and saying, "TOO DIFFERENT FROM REALITY!"
Other folks don't do that, and while not 1:1, they are able to correlate the performance of a benchmark with the performance of their own use case.
Try not to get wrapped around the axle on "everyone", by the way, it's not literal.
Similar to the phrase "Lies, damned lies, and Benchmarks" , should be the phrase "Lies, damned lies, and AWS costs"
the past few years at memsql we have watched quietly as competitors have released numerous benchmarks. often one particular benchmark that coincidentally fits into a niche strength. often cleverly ignoring its weaknesses. often against an antiquated version of our software.
we are quite proud to be the only database company to produce such a varied array of impressive benchmark results. no niche benchmarks. no old versions. as close to the truth as it gets.
other databases in the mirror may be further behind than they appear
This benchmark is pretty ridiculous for the following reasons: 1. Their database is run in asynchronous durability mode. 2. They specifically do the one thing that TPC-C says you shouldn't do, which is get really high throughput on a small dataset. TPC-C enforces that you scale your data-stored with the query throughput. CockroachDB maxes out at ~12.8tpmC/warehouse because its waiting at the legal maximum throughput…
1. MemSQL is running with synchronous replication in all these benchmarks. All data is stored on a 2nd machine before any transaction is acknowledged as committed. You’re right this is not as strong as running with both synchronous writes to disk and over the network. MemSQL supports this as well and results in about a 40 to 50% performance hit depending on the disk speed. Very few of our customers run in this configuration so we didn’t include it (the edge case of multiple machines losing power is not worth the performance hit for them).
2. Can you point me to what you’re describing in the TPC-C specification? I have never heard of what you’re claiming. TPC-C has maximum allowed latency requirements for the 5 transaction types it runs and also requirements around the mix of those transactions in the workload. The goal of the benchmark is still to run as many "New Order" transaction per minute while maintaining the latency requirements of the other unmeasured transactions running in the background (this is what tpmC stands for). We used the Percona TPC-C driver for MySQL to handle this (with a few small bug fixes).
3. The main thing we wanted to show is that our performance on TPC-DS is similar (better at some scale factors, slower on others) to data warehouses that specialize in running these types of queries. We likely should have provided more details (per query break downs and what not).
4. We used the Percona MySQL TPC-C driver with some changes to make the initial data loading faster. That driver uses the “FOR UPDATE” clause in MySQL instead of running in serializable isolation level.
I know you did a lot of work on CockroachDB. The point of the blog post was not to attack cockroach (I personally didn’t want to mention it at all), but to show how MemSQL is different. We are one of the few distributed SQL databases with competitive results on all 3 major TPC benchmarks.
Earlier quoted context omitted.
> The problem you're having I'm not having a problem. > is that everybody already knows that, Certainly, some people act like they don't, at least in the context of specific benchmarks. And some people outright claim the opposite of what you say everyone understands, e.g., by claiming that benchmarks inherently correlate with every possible use of a technology. > and plenty of folks can still extract value out of a b…
The problem you're having is you can't figure out how to use benchmarks to understand how a system works. You're throwing up your hands and saying, "TOO DIFFERENT FROM REALITY!" Other folks don't do that, and while not 1:1, they are able to correlate the performance of a benchmark with the performance of their own use case. Try not to get wrapped around the axle on "everyone", by the way, it's not literal.
You can correlate lots of things. Poor countries increase penis size. Ice cream leads to murder. Cheese kills people by tangling them in their bedsheets. These things are strongly correlated. But they have no causative relationship.
To put it another way, a single benchmark test used find a solution is scientifically equivalent to finding one "average" man, making him run a series of athletic tests, and making his performance the benchmark for men's athletics.