Live data from Hacker News

How MySQL is able to scale to 200M QPS – MySQL Cluster

highscalability.com

11–20 of 25 posts

Re: How MySQL is able to scale to 200M QPS – MySQL Cluster

#11
post #7
post #4

> distributed transactions based on 2-phase commit. i just stop reading from there.

That's the way Megastore does it too. Optional 2PC for transactions that span entity groups. So what's wrong with 2PC?

If you have a distributed system and need to create a transaction across multiple nodes 2PC has some shortcomings[1], the 3PC is addressing this.

[1] https://en.wikipedia.org/wiki/Three-phase_commit_protocol#Mo...

Re: How MySQL is able to scale to 200M QPS – MySQL Cluster

#12

A modern and properly implemented database engine should be bottlenecked on network saturation for these types of workloads even if backed by disk storage, never mind in-memory. These workloads are trivially distributable, so scale-out is expected unless you are doing something really wrong. In that sense, similar "look how fast my simple lookups are" benchmarks published by other database vendors are really a functi…

I'd be interested to see a benchmark test suite for ACID-compliant databases with, as you mentioned, complex constraints and joins. Get as close to real-world workloads as possible.

I don't want to see the quarter mile, I want to see you road rally.

Re: How MySQL is able to scale to 200M QPS – MySQL Cluster

#16

> MySQL Cluster is a scalable, real-time in-memory, ACID-compliant transactional database To what degree can something be called durable if it's in-memory? > It is possible to choose how to store data; either all in memory or with some on disk (non-indexed data only). […] Disk-based data can be used to store data with less strict performance requirements, where the data set is larger than the available RAM. It's grea…

It's durable using checkpointing and redologging, this is default behavior. You can read more details here: http://www.clusterdb.com/mysql-cluster/how-can-a-database-be... And I agree that using the name MySQL is a bit misleading, perhaps better with NDB (NetworkDataBase) as NDB only uses MySQL as an API for SQL access to data.

Re: How MySQL is able to scale to 200M QPS – MySQL Cluster

#17

A modern and properly implemented database engine should be bottlenecked on network saturation for these types of workloads even if backed by disk storage, never mind in-memory. These workloads are trivially distributable, so scale-out is expected unless you are doing something really wrong. In that sense, similar "look how fast my simple lookups are" benchmarks published by other database vendors are really a functi…

There are some more complex benchmarks here: http://www.mysql.com/why-mysql/benchmarks/mysql-cluster/ Best way as always is by running benchmark on your specific workload, there is lot's of help to be found here: http://mikaelronstrom.blogspot.se/ if you want to get started benchmarking MySQL Cluster.

Re: How MySQL is able to scale to 200M QPS – MySQL Cluster

#18

> MySQL Cluster is a scalable, real-time in-memory, ACID-compliant transactional database To what degree can something be called durable if it's in-memory? > It is possible to choose how to store data; either all in memory or with some on disk (non-indexed data only). […] Disk-based data can be used to store data with less strict performance requirements, where the data set is larger than the available RAM. It's grea…

VoltDB replicates to k nodes so you can choose the level of durability. Combined with UPSes in separate parts of the datacenter, it's durable in some senses. (I'd be more concerned with a kernel or app bug that crashes the system, if there wasn't a "save RAM on unclean boot". This only applies to the commercial version, though. The open source one ditched that feature.

VoltDB, like MySQL Cluster and several other in-memory system, uses snapshot checkpoints and logging to maintain power-loss protection. VoltDB is a bit different because of how it uses determinism + logical logging, but the point is that most of these systems don't have to lose data on power loss.

Re: How MySQL is able to scale to 200M QPS – MySQL Cluster

#19

I'm sorry but MySQL Cluster is pure refined salty horse shit. Nothing more. We're currently at the office trying to recover our production data from its filthy hands.

>>>> pure refined salty horse shit. Nothing more

Why did you deployed such a "horse shit" product ?

>>>> to recover our production data

Don't you have a backup ?

Re: How MySQL is able to scale to 200M QPS – MySQL Cluster

#20
post #11
post #7

Earlier quoted context omitted.

That's the way Megastore does it too. Optional 2PC for transactions that span entity groups. So what's wrong with 2PC?

If you have a distributed system and need to create a transaction across multiple nodes 2PC has some shortcomings[1], the 3PC is addressing this. [1] https://en.wikipedia.org/wiki/Three-phase_commit_protocol#Mo...

Note that the 2PC in Ndb is not quite the same as that used for the normal '2PC is blocking' arguments :

  1.  The transaction coordinator (TC) role is 
      automatically taken over in case of the 
      failure of the coordinator during commit.  
      The new coordinator uses the surviving 
      participants' states to quickly decide 
      on Commit or Abort so that locked 
      resources are released.

  2.  As the participants are part of an integrated 
      system, failure of both the coordinator and 
      e.g. the only participant to be aware of the 
      commit decision can be handled correctly.  
      In this case the transaction is quickly 
      aborted by the new coordinator and on recovery 
      the old participant will 'undo' its local
      committed-state before rejoining the cluster.

  3.  Participants' local commit decisions are not 
      coupled directly to lock release / visibility 
      of those decisions.  Locks are released only 
      when the participant-failure-durability of 
      the commit decision is assured (e.g. all 
      replicas have committed).

  4.  Ndb has a third phase (The complete phase) 
      which releases locks on replicas.  This is 
      not the same as the algorithm referred to 
      as three phase commit, so we don't call it 
      three phase commit.  Also, the third Complete 
      phase is not part of the critical commit 
      path - the commit acknowledgement is sent 
      to the application prior to the start of 
      the third phase, so it does not affect 
      the user's experienced commit latency.
Not all '2PC' implementations are created equal.
Post reply on HN