Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Bunch of Money on AWS and Some Benchmark Results

memsql.com

Post reply on HN
Print view View original

11–20 of 44 posts

Re: A Bunch of Money on AWS and Some Benchmark Results

#11

Post by [deleted] »

AdamProut wrote: post #6
tyingq wrote: post #2

Might be advisable to note that you don't support foreign keys if you're going to show how much better your performance is versus a database that does.

(MemSQL CTO here) You're right, MemSQL doesn't support foreign keys as of yet, but none of these benchmarks require foreign key support. Two of them (TPC-H and TPC-DS) are a set of complex SELECT queries where foreign keys are not relevant at all. TPC-C is a write heavy benchmark, but the specification doesn't require foreign keys to be maintained (the data model does indicate the foreign key relationships though)[1]…

[deleted]

Re: A Bunch of Money on AWS and Some Benchmark Results

#12

Post by nikita »

AdamProut wrote: post #8
scarface74 wrote: post #5

Earlier quoted context omitted.

It’s an OLAP database. Are you really that concerned about FK constraints? The only other OLAP database that I’ve used is Amazon’s Redshift and FKs are for “informational purposes”. Or is it considered a transactional database or an analytics database?

MemSQL can run both types of workloads. That is one the main things we're trying to demonstrate by running one of the most popular operational database benchmarks (TPC-C) as well as the two most popular analytics benchmarks (TPC-DS, TPC-H). Our strength is somewhere in the middle[1] [1] https://www.memsql.com/blog/the-need-for-operational-analyti...

Another way to think about MemSQL is that it's an operational database. This means that it can support high velocity transactions and power apps. The fact it can run decision support (OLAP) benchmarks is an important differentiation. Since those benchmarks are SQL benchmarks you can think of it as simple as "it's just great in SQL". Or you can think about it as "if my app has reporting as part of the workloads MemSQL will handle it out of the box". Finally there are many use cases that fall in the category of "Operational Analytics": time series, IoT, real-time dashboarding, moving from batch to streaming, predictive maintenance. All of those require underlying infrastructure to handle transactions and complex SQL in one system.

Re: A Bunch of Money on AWS and Some Benchmark Results

#13

Post by tyingq »

AdamProut wrote: post #6
tyingq wrote: post #2

Might be advisable to note that you don't support foreign keys if you're going to show how much better your performance is versus a database that does.

(MemSQL CTO here) You're right, MemSQL doesn't support foreign keys as of yet, but none of these benchmarks require foreign key support. Two of them (TPC-H and TPC-DS) are a set of complex SELECT queries where foreign keys are not relevant at all. TPC-C is a write heavy benchmark, but the specification doesn't require foreign keys to be maintained (the data model does indicate the foreign key relationships though)[1]…

Just seems like you would want that noted if you're going to compare with CockroachDB and say things like "Our results show that we can do both transaction processing and data warehousing well".

Re: A Bunch of Money on AWS and Some Benchmark Results

#14

Post by sroussey »

georgewfraser wrote: post #9

As someone who's spent a lot of time working with TPC-DS [1] and talking to people about it [2], I see a couple areas that could be improved in this benchmark: 1. Total run time is not an appropriate way to summarize the performance across queries, because some queries take 100x longer than others. The appropriate way to summarize this kind of data is to use the geomean [3]. 2. The official TPC-DS queries make heavy…

Curious to see TiDB in there

Re: A Bunch of Money on AWS and Some Benchmark Results

#15

Post by AdamProut »

sroussey wrote: post #14
georgewfraser wrote: post #9

As someone who's spent a lot of time working with TPC-DS [1] and talking to people about it [2], I see a couple areas that could be improved in this benchmark: 1. Total run time is not an appropriate way to summarize the performance across queries, because some queries take 100x longer than others. The appropriate way to summarize this kind of data is to use the geomean [3]. 2. The official TPC-DS queries make heavy…

Curious to see TiDB in there

You need columnstore storage and a reasonably mature query optimizer to get good results on TPC-DS. TiDB is lacking both right now.

Re: A Bunch of Money on AWS and Some Benchmark Results

#16

Post by arjunnarayan »

This benchmark is pretty ridiculous for the following reasons:

1. Their database is run in asynchronous durability mode.

2. They specifically do the one thing that TPC-C says you shouldn't do, which is get really high throughput on a small dataset. TPC-C enforces that you scale your data-stored with the query throughput. CockroachDB maxes out at ~12.8tpmC/warehouse because its waiting at the legal maximum throughput, as opposed to running up the numbers in a way that's against the rules (and spirit) of the benchmark.

3. They make all the TPC-DS mistakes that georgewfraser points out elsewhere in this thread.

4. They run in read committed mode (they don't support anything higher), CockroachDB runs in serializable mode.

I ended up ranting about this on Twitter, so rather than reproducing everything here, I'm going to link to my rant there. Apologies for the cross-posting across fora: https://twitter.com/narayanarjun/status/1128393193941274624

Re: A Bunch of Money on AWS and Some Benchmark Results

#18

Post by diminoten »

Regardless of the minor technical nitpicks here, I respect the hell out of the person/people who wrote this article, if only because I know how insanely hard it is to express a technical thought in great detail while maintaining a certain amount of levity to try and keep people interested.

Re: A Bunch of Money on AWS and Some Benchmark Results

#19

Post by diminoten »

peterwwillis wrote: post #7
wmf wrote: post #4

Earlier quoted context omitted.

Because?

Because all a benchmark tells you is the result of one use case. It's not an estimate of what your use will result in. It's basically a sales pitch. And sadly, upper management falls for it all the time, often without an evaluation period or even spitballing it for a week with the teams that would be using it. Tons of factors will change the result for a particular use case, regardless of how "normalized" the test tr…

The whole point of benchmarks is that their "use case" correlates roughly with every use case related to the technology being benchmarked.

Is the correlation 1:1? No. Is it still relevant when making a decision about what technology to use? Absolutely.

Re: A Bunch of Money on AWS and Some Benchmark Results

#20

Post by dongxu »

AdamProut wrote: post #15
sroussey wrote: post #14

Earlier quoted context omitted.

Curious to see TiDB in there

You need columnstore storage and a reasonably mature query optimizer to get good results on TPC-DS. TiDB is lacking both right now.

TiDB developer here, we're working on columnstore for TiDB, will release in the near future. On the query optimizer side, I think in TiDB 3.0, TiDB's optimizer has given the best execution plan for most of the TPC-H (TPC-H 50G) queries. But we've never tested on TPC-DS yet.
Post reply on HN