A Bunch of Money on AWS and Some Benchmark Results • Hacker News Board

Re: A Bunch of Money on AWS and Some Benchmark Results

#41

Post by dkhenry » Wed, May 15, 2019, 2:09 AM UTC

knz42 wrote: post #40

dkhenry wrote: post #39

Earlier quoted context omitted.

That "Think Time" that you are referring to is supposed to emulate users running transactions on the database. So its not the database waiting its the driver waiting. While I do understand the reason for putting that in, you know very well that violating that limit doesn't artificially give CockroachDB or MemSQL an advantage when you are talking about 100,000 warehouses and a random distribution off transactions. If…

You should read up on why this think time exists. It has nothing to do with "emulating slow clients" and everything to do with not claiming "I have a fast database" by running gazillion txn/sec on just 1MB of data in RAM. TPC-C requires that you increase the amount of "live data" if you want to display/advertise more performance. That's the benchmark's rule. If you want to benchmark something else, that's fine, but t…

I know why it exists, and my point is they are not trying to show a gazillion txn/sec on 1MB of data. You are looking at a dataset that is several TB's. They have far surpassed the point where a vendor is trying to cheat by putting all the data in L1 cache and claiming to be fast.

Re: A Bunch of Money on AWS and Some Benchmark Results

#42

Post by bishala » Wed, May 15, 2019, 3:11 AM UTC

Shorn wrote: post #35

arjunnarayan wrote: post #16

This benchmark is pretty ridiculous for the following reasons: 1. Their database is run in asynchronous durability mode. 2. They specifically do the one thing that TPC-C says you shouldn't do, which is get really high throughput on a small dataset. TPC-C enforces that you scale your data-stored with the query throughput. CockroachDB maxes out at ~12.8tpmC/warehouse because its waiting at the legal maximum throughput…

What's the logic behind linking from HN to twitter? I would've though it'd be easier to write on HN (and I know for a fact that it's painful for me to read on Twitter). This is a genuine question by the way. I followed the link, then couldn't be bothered to try and make sense of it and got to wondering why you would do that.

He said that he doesn't want to reiterate what he already said on Twitter. That twitter thread is quite a few tweets long so I think it makes sense to just link to it.

Re: A Bunch of Money on AWS and Some Benchmark Results

#43

Post by AdamProut » Wed, May 15, 2019, 5:36 AM UTC

arjunnarayan wrote: post #36

AdamProut wrote: post #28

Earlier quoted context omitted.

(MemSQL CTO here) 1. MemSQL is running with synchronous replication in all these benchmarks. All data is stored on a 2nd machine before any transaction is acknowledged as committed. You’re right this is not as strong as running with both synchronous writes to disk and over the network. MemSQL supports this as well and results in about a 40 to 50% performance hit depending on the disk speed. Very few of our customers…

1. Comparing numbers from one system (Cockroach) that adheres to strict durability requirements to another that does not (MemSQL) is apples to oranges, especially, as you point out, you see a 2x performance hit when you impose that requirement. 2. What you're looking for is the 'Think Time' mentioned in the TPC-C spec[1] (table in 5.2.5.7). From 5.2.5.2, I quote: > for each transaction type, the Keying Time is consta…

Thank you for the the details. Its pretty clear at this point that its not a fair comparison. The TPC-C driver we used (Percona's MySQL TPC-C driver) is pushing MemSQL as hard as it can and isn't following the "think time" part of the spec that artificially slows the driver down. So, this understandably gives us higher throughput numbers. We'll remove the comparison and make it clear the driver we are using isn't obeying the "think time" part of the spec.

Again, our goal here is not to have some showdown with cockroach. We don't really compete with each other. Our goal is to show the breadth of workloads MemSQL can run (fast in-memory point queries as well complex OLAP queries over large tables). None the less, we should have caught this before we published the article. I appreciate the correction.

Re: A Bunch of Money on AWS and Some Benchmark Results

#44

Post by mc110 » Wed, May 15, 2019, 9:03 AM UTC

The great thing about AWS and other cloud platforms is that you can put the data in e.g. S3, then use the tool of choice for your workloads.

I'm assuming that MemSQL works fine with that sort of configuration, rather than requiring you to lock your data up in some proprietary format.

Also, unlike the bad old days of on-premise platforms, you can try things out to see how they work. You could even do that with a public dataset first, to see how it works (see https://registry.opendata.aws/ for a list of these).

For example, there is an Amazon Customer Review dataset of over 160 million customer reviews - you could use that and try MemSQL for various use cases, then look at alternatives.

Disclaimer - clearly as a Kognitio employee I'd suggest you looked at us for analytics use cases, and you can see an example of sentiment analysis as scale using the Amazon Customer Review data set at https://kognitio.com/blog/sentiment-analysis-amazon-reviews-.... Also, a couple of articles on LinkedIn at https://www.linkedin.com/pulse/100-shades-grey-other-amazon-... for another piece of work on that same data, and https://www.linkedin.com/pulse/media-brexit-story-so-far-may... for a view on Global Media coverage of Brexit over time.