A Bunch of Money on AWS and Some Benchmark Results
1–10 of 44 posts
Re: A Bunch of Money on AWS and Some Benchmark Results
#2Might be advisable to note that you don't support foreign keys if you're going to show how much better your performance is versus a database that does.
Re: A Bunch of Money on AWS and Some Benchmark Results
#3Similar to the phrase "Lies, damned lies, and Benchmarks", should be the phrase "Lies, damned lies, and AWS costs"
Re: A Bunch of Money on AWS and Some Benchmark Results
#4Similar to the phrase "Lies, damned lies, and Benchmarks" , should be the phrase "Lies, damned lies, and AWS costs"
Because?
Re: A Bunch of Money on AWS and Some Benchmark Results
#5Might be advisable to note that you don't support foreign keys if you're going to show how much better your performance is versus a database that does.
It’s an OLAP database. Are you really that concerned about FK constraints?
The only other OLAP database that I’ve used is Amazon’s Redshift and FKs are for “informational purposes”.
Or is it considered a transactional database or an analytics database?
Re: A Bunch of Money on AWS and Some Benchmark Results
#6Might be advisable to note that you don't support foreign keys if you're going to show how much better your performance is versus a database that does.
(MemSQL CTO here)
You're right, MemSQL doesn't support foreign keys as of yet, but none of these benchmarks require foreign key support. Two of them (TPC-H and TPC-DS) are a set of complex SELECT queries where foreign keys are not relevant at all. TPC-C is a write heavy benchmark, but the specification doesn't require foreign keys to be maintained (the data model does indicate the foreign key relationships though)[1].
These are unofficial benchmark results (not independently verified by TPC), so our interpretation of the specs may not be 100% correct, but I think we got it right as far as foreign keys are concerned.
[1] http://www.tpc.org/tpc_documents_current_versions/pdf/tpc-c_...
Re: A Bunch of Money on AWS and Some Benchmark Results
#7Similar to the phrase "Lies, damned lies, and Benchmarks" , should be the phrase "Lies, damned lies, and AWS costs"
Because?
Because all a benchmark tells you is the result of one use case. It's not an estimate of what your use will result in. It's basically a sales pitch. And sadly, upper management falls for it all the time, often without an evaluation period or even spitballing it for a week with the teams that would be using it. Tons of factors will change the result for a particular use case, regardless of how "normalized" the test tries to be (especially considering most are designed for high-load high-performance high-scale scenarios, which isn't the average real-world use case; most people just have unoptimized queries, or a shit legacy stack, or they expect to eventually have a big use case and were told to find either the fastest or cheapest solution, which may not even be in AWS to begin with, and so on...). The AWS cost is just another benchmark, so it's just as much a lie.
Re: A Bunch of Money on AWS and Some Benchmark Results
#8Might be advisable to note that you don't support foreign keys if you're going to show how much better your performance is versus a database that does.
It’s an OLAP database. Are you really that concerned about FK constraints? The only other OLAP database that I’ve used is Amazon’s Redshift and FKs are for “informational purposes”. Or is it considered a transactional database or an analytics database?
MemSQL can run both types of workloads. That is one the main things we're trying to demonstrate by running one of the most popular operational database benchmarks (TPC-C) as well as the two most popular analytics benchmarks (TPC-DS, TPC-H). Our strength is somewhere in the middle[1]
[1] https://www.memsql.com/blog/the-need-for-operational-analyti...
Re: A Bunch of Money on AWS and Some Benchmark Results
#9As someone who's spent a lot of time working with TPC-DS [1] and talking to people about it [2], I see a couple areas that could be improved in this benchmark:
1. Total run time is not an appropriate way to summarize the performance across queries, because some queries take 100x longer than others. The appropriate way to summarize this kind of data is to use the geomean [3].
2. The official TPC-DS queries make heavy use of grouping sets, which are a rarely-used SQL feature. I think TPC-DS is better if you rewrite the queries to eliminate grouping sets.
3. You used the exact same queries to "warm up" the data warehouse, and to test the performance. Some data warehouses (notably Redshift) aggressively cache intermediate compilation results, so they are much faster the second time they see a query or even a fragment of a query. To model a real user submitting queries interactively, you should use warmup queries that are similar to but not the same as the ones you use to measure performance.
4. You can solve the "vendor benchmarking their own product" problem by submitting a PR to our repo [4], which currently tests Redshift, Snowflake, BigQuery, Azure SQL DW, and Presto. We'd be happy to review it and endorse the timing if it meets our standards for fairness!
[1] https://fivetran.com/blog/warehouse-benchmark
[2] https://www.youtube.com/watch?v=XpaN-PqSczM
Re: A Bunch of Money on AWS and Some Benchmark Results
#10Might be advisable to note that you don't support foreign keys if you're going to show how much better your performance is versus a database that does.
It’s an OLAP database. Are you really that concerned about FK constraints? The only other OLAP database that I’ve used is Amazon’s Redshift and FKs are for “informational purposes”. Or is it considered a transactional database or an analytics database?
[deleted]