A Bunch of Money on AWS and Some Benchmark Results

Again, I'm not having a problem.

> is you can't figure out how to use benchmarks to understand how a system works

No, I have no partucular problem evaluating whether a benchmark is useful to a decision and if so, how.

Nor have I said anything indicating any such problem.

> You're throwing up your hands and saying, "TOO DIFFERENT FROM REALITY!"

No, I'm saying he naive statement upthread that benchmark performance correlates with every possible use of a technology is nonsense.

That's it.

> Other folks don't do that

Actually, some do, but that's neither here nor there.

> and while not 1:1, they are able to correlate the performance of a benchmark with the performance of their own use case.

Once again, yes, lots of people have the skill to figure out whether and in what way particular benchmarks have utility for their usecases. I explicitly said that in the post you are responding to.

That's very different than your claim that I reacted against, which is that any benchmark inherently correlates with every use case, which is—again—complete nonsense.

Oh. That's what you thought I said? Right. I didn't mean to say that, and if somehow I did say that I withdraw. Benchmarks are valuable if you know how to use them, but of course I agree with you, a benchmark isn't always relevant to every use case.

Side note, quoting many small portions of a person's comment and replying exclusively to the quoted bits is inferior to replying to them with full sentences/paragraphs. I've only ever seen what you're doing done by folks who are super interested in arguing and completely uninterested in having a conversation.

It definitely comes across like you do have a problem, which you have repeatedly stated you do not (and I believe you)! It just muddies the water, what you're doing here.

2. What you're looking for is the 'Think Time' mentioned in the TPC-C spec[1] (table in 5.2.5.7). From 5.2.5.2, I quote:

> for each transaction type, the Keying Time is constant and must be a minimum of 18 seconds for New- Order, 3 seconds for Payment, and 2 seconds each for Order-Status, Delivery, and Stock-Level.

Chapter 4 is pretty thorough on elaborating on this. The comment under section 4.1.3 explicitly states:

> Comment: The maximum throughput is achieved with infinitely fast transactions resulting in a null response time and minimum required wait times. The intent of this clause is to prevent reporting a throughput that exceeds this maximum, which is computed to be 12.86 tpmC per warehouse.

Again, CockroachDB numbers are right up against this limit - because the database is waiting, as required! It's within ~99% of the maximum allowed. No bar is allowed to go more than 1% higher! So stacking a bar chart next to it that goes 10x higher is pretty misleading.

3. I'm pretty impressed that you can run all the TPC-DS queries. That's pretty impressive. But performance wise, there really isn't enough fleshed out, and given that the TPC-DS authors explicitly disavow the single metric that you use (power test numbers), is simply too little to claim parity to existing databases. That said, in this conversation I'm an OLTP guy; I'll let others more experienced with Data Warehouse benchmarking take this up, e.g. [3]

4. This one I'll concede that you are doing the appropriate thing as per spec (SELECT FOR UPDATE ensures serializability), but it's the single part of the spec that's not held up over time - the paper "Making Snapshot Isolation Serializable" is a great explanation of just what lengths you have to go to to prove that a set of transactions only provide serializable histories when run in a degraded isolation mode. That said, fair enough, no anomalies will be present due to Alan Fekete's proof. But do note that CockroachDB is doing a lot of extra work (work that MemSQL can elide, since it's simply not checking for isolation anomalies) to ensure that histories are always serializable[4].

5. While I don't work there, I did a lot of work specifically on benchmarking CockroachDB, and would like to politely request that you take down those bars for CockroachDB, since you're taking numbers that are shackled to the THINK TIME maximum and comparing them to a system that is not.

[1]: http://www.tpc.org/tpc_documents_current_versions/pdf/tpc-c_...

[2]: https://dl.acm.org/citation.cfm?id=1071615

[3]: https://twitter.com/gregrahn/status/1128448156180422656

[4]: I'll shamelessly plug my blog post on this for the reader interested in more about transaction isolation levels: https://ristret.com/s/f643zk/history_transaction_histories

> How come you haven't go to 10T?

We don't see a lot of 10 TB single datasets "in the wild". We think 100 GB - 1 TB is more representative of the average real-world data warehouse user.

> for us the warmup time separates the time to compile and code gen

You should include compile time---your compile time is really good! Redshift sometimes takes longer to compile the query than to run it, though I think they may have improved this in the last 6 months.

TPC-C requires that you increase the amount of "live data" if you want to display/advertise more performance. That's the benchmark's rule.

If you want to benchmark something else, that's fine, but then

1) don't call it "TPC-C" 2) don't compare with databases that play by the rules.