Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Achieved 40K QPS and 20B Rows of Data in a Single Scale-Out Cluster

pingcap.com

Post reply on HN
Print view View original

11–19 of 19 posts

Re: How We Achieved 40K QPS and 20B Rows of Data in a Single Scale-Out Cluster

#11

Post by jiggawatts »

Caitin_Chen wrote: post #3
rubyn00bie wrote: post #2

Nice article, I just wish they went into more detail as to why TiKV was chosen over other distributed stores like: Cockroach, Yugabyte, or Foundation. It'd also be nice to know the size of those 40bn rows is it 10TB or 100TB?

For database selection, there is a blog post about why choose TiDB over CockroachDB and other MySQL-based solutions: https://pingcap.com/success-stories/why-we-chose-a-distribut...

It's interesting to see how crazy complex some of these scale-out solutions end up being, especially when they start off with something like MySQL and are forced to maintain compatibility while scaling far beyond what it was designed to do.

You've got to wonder if simply sharding by either player or game would have worked well enough, even with MySQL. Worked well enough for Blizzard for over a decade!

Similarly, I wonder if a single active write server paired with a handful of readable secondaries could have taken this query volume in its stride if using something like MS SQL Server on high-spec bare metal kit. Think In-Memory processing on dual AMD EPYC 2 servers with 128 cores per box.

They mention billions of rows and terabytes of data, but I've heard of similar scale systems as far back as over decade ago! The Australian phone company Telstra did all their billing in a single IBM DB2 database, for example, and the incoming data in that system is just as real time.

I'm probably wrong, but this smells like the company missed some basic optimisation opportunities somewhere...

Re: How We Achieved 40K QPS and 20B Rows of Data in a Single Scale-Out Cluster

#12

Post by adev_ »

jiggawatts wrote: post #11
Caitin_Chen wrote: post #3

Earlier quoted context omitted.

For database selection, there is a blog post about why choose TiDB over CockroachDB and other MySQL-based solutions: https://pingcap.com/success-stories/why-we-chose-a-distribut...

It's interesting to see how crazy complex some of these scale-out solutions end up being, especially when they start off with something like MySQL and are forced to maintain compatibility while scaling far beyond what it was designed to do. You've got to wonder if simply sharding by either player or game would have worked well enough, even with MySQL. Worked well enough for Blizzard for over a decade! Similarly, I wo…

> You've got to wonder if simply sharding by either player or game would have worked well enough, even with MySQL. Worked well enough for Blizzard for over a decade!

With all the respect I have for blizzard, I would not take old Blizzard as an example of scalability handling. I remember a time where there authentication system was well known to give up under load after every update.

Re: How We Achieved 40K QPS and 20B Rows of Data in a Single Scale-Out Cluster

#13

Post by jinqueeny »

jinqueeny wrote: post #6
chucky_z wrote: post #4

Calling cockroachdb "not open source" is a stretch. https://github.com/cockroachdb/cockroach (The title is also wildly inaccurate to what's in the article, and should just have the articles title)

Thanks for the comment! CockroachDB is now under BSL and CCL licenses. Different users have different opinions but you are right, we need to be accurate about it. Please give me some time to consult our user and see whether we can update this case study.

This case study has been updated: https://pingcap.com/success-stories/why-we-chose-tidb-over-o...

Re: How We Achieved 40K QPS and 20B Rows of Data in a Single Scale-Out Cluster

#14

Post by _tkzm »

dongxu wrote: post #10
_tkzm wrote: post #7

Pingcap is loosing hundreds and thousands of potential customers and millions of dollars by not having a single-binary product. The current setup is over-complicated and confusing and I would not touch it with a 10ft pole, as they say. That's why it's worth rewriting the app from mysql syntax to postgre syntax and use cockroachdb more than spending dev , ops , cloud* hours on making legacy code work with this thing.…

PingCAP CTO here, thanks for these comments! We highly appreciate all the feedback! First, it’s true that the current setup/deployment of TiDB is not easy. This is something that we're making serious moves to improve. For example, A. We provide Ansible playbooks to simplify the deployment and rolling upgrade for on-prem users; B. We built and open-sourced TiDB Operator ( https://github.com/pingcap/tidb-operator ) to…

all those points sound more like quickfixes for bad architecture design. I've been there, done that.. and i've learnt from it. instead of fixing the initial problem, you are just throwing more code and complexity over it. which costs you money and time. i think you made a BIG mistake byt going polyglot and not sticking to single a language which now prevents you from merging the code into a single binary in cost-efficient way.

if you decide to go with the single binary approach, i am curious to see if you decide to go with Go or Rust :)

just fyi: cdb went with Go but their storage layer ended up being rewritten with C, so they too are not much different from you, with the exception of being able to run a single binary via Cgo which you cannot do with Rust.

Re: How We Achieved 40K QPS and 20B Rows of Data in a Single Scale-Out Cluster

#15

Post by PeterZaitsev »

Caitin_Chen wrote: post #3
rubyn00bie wrote: post #2

Nice article, I just wish they went into more detail as to why TiKV was chosen over other distributed stores like: Cockroach, Yugabyte, or Foundation. It'd also be nice to know the size of those 40bn rows is it 10TB or 100TB?

For database selection, there is a blog post about why choose TiDB over CockroachDB and other MySQL-based solutions: https://pingcap.com/success-stories/why-we-chose-a-distribut...

The headline Performance numbers better speak for "when not to use sharding" use case. 40K QPS is trivial for even medium size MySQL Deployments with high end servers being able to handle 1M QPS (simple queries)

20Bil rows is also quite trivial - with 100 Bytes rows this will be just 2TB of data.

We probably look at long rows in this case and complicated queries but as they read numbers are rather unimpressive :)

Re: How We Achieved 40K QPS and 20B Rows of Data in a Single Scale-Out Cluster

#16

Post by gravypod »

_tkzm wrote: post #14
dongxu wrote: post #10

Earlier quoted context omitted.

PingCAP CTO here, thanks for these comments! We highly appreciate all the feedback! First, it’s true that the current setup/deployment of TiDB is not easy. This is something that we're making serious moves to improve. For example, A. We provide Ansible playbooks to simplify the deployment and rolling upgrade for on-prem users; B. We built and open-sourced TiDB Operator ( https://github.com/pingcap/tidb-operator ) to…

all those points sound more like quickfixes for bad architecture design. I've been there, done that.. and i've learnt from it. instead of fixing the initial problem, you are just throwing more code and complexity over it. which costs you money and time. i think you made a BIG mistake byt going polyglot and not sticking to single a language which now prevents you from merging the code into a single binary in cost-effi…

> all those points sound more like quickfixes for bad architecture design.

As an on looker the architecture of TiDB is anything but bad. It doesn't fit into the "run one thing for everything" bucket of software because it's designed to be horizontally scalable. Every part of the design is scoped well enough to only do the bare minimum for it's goal. I have no doubt that internally at Google things like BigTable look very architecturally similar.

> Big mistake going polyglot.

Why? Because you can't make a single binary? That's not really a use case that makes sense for a cloud native DB. Everything is going to be a container anyway, so it doesn't matter what artifacts are included. Also the individual implementation layers of TiDB are individually useful (TiKV).

Making a single binary has no benifits unless you're running your hardware in a pet mentality which manually curated software. In this case a horizontally scalable db will not help you.

Also, providing an operator is a far cry from a "quick fix". I applaud them for doing this because it essentially removes the operation burden of running their DB.

Re: How We Achieved 40K QPS and 20B Rows of Data in a Single Scale-Out Cluster

#17

Post by _tkzm »

gravypod wrote: post #16
_tkzm wrote: post #14

Earlier quoted context omitted.

all those points sound more like quickfixes for bad architecture design. I've been there, done that.. and i've learnt from it. instead of fixing the initial problem, you are just throwing more code and complexity over it. which costs you money and time. i think you made a BIG mistake byt going polyglot and not sticking to single a language which now prevents you from merging the code into a single binary in cost-effi…

> all those points sound more like quickfixes for bad architecture design. As an on looker the architecture of TiDB is anything but bad. It doesn't fit into the "run one thing for everything" bucket of software because it's designed to be horizontally scalable. Every part of the design is scoped well enough to only do the bare minimum for it's goal. I have no doubt that internally at Google things like BigTable look…

https://www.youtube.com/watch?v=pWdd6_ZxX8c

Re: How We Achieved 40K QPS and 20B Rows of Data in a Single Scale-Out Cluster

#18

Post by steveklabnik »

_tkzm wrote: post #14
dongxu wrote: post #10

Earlier quoted context omitted.

PingCAP CTO here, thanks for these comments! We highly appreciate all the feedback! First, it’s true that the current setup/deployment of TiDB is not easy. This is something that we're making serious moves to improve. For example, A. We provide Ansible playbooks to simplify the deployment and rolling upgrade for on-prem users; B. We built and open-sourced TiDB Operator ( https://github.com/pingcap/tidb-operator ) to…

all those points sound more like quickfixes for bad architecture design. I've been there, done that.. and i've learnt from it. instead of fixing the initial problem, you are just throwing more code and complexity over it. which costs you money and time. i think you made a BIG mistake byt going polyglot and not sticking to single a language which now prevents you from merging the code into a single binary in cost-effi…

You can do the "single binary" thing with Rust if you use musl.

Re: How We Achieved 40K QPS and 20B Rows of Data in a Single Scale-Out Cluster

#19

Post by irfansharif »

_tkzm wrote: post #14
dongxu wrote: post #10

Earlier quoted context omitted.

PingCAP CTO here, thanks for these comments! We highly appreciate all the feedback! First, it’s true that the current setup/deployment of TiDB is not easy. This is something that we're making serious moves to improve. For example, A. We provide Ansible playbooks to simplify the deployment and rolling upgrade for on-prem users; B. We built and open-sourced TiDB Operator ( https://github.com/pingcap/tidb-operator ) to…

all those points sound more like quickfixes for bad architecture design. I've been there, done that.. and i've learnt from it. instead of fixing the initial problem, you are just throwing more code and complexity over it. which costs you money and time. i think you made a BIG mistake byt going polyglot and not sticking to single a language which now prevents you from merging the code into a single binary in cost-effi…

> cdb went with Go but their storage layer ended up being rewritten with C,

The storage layer was not rewritten in C, it's just RocksDB. There's ongoing work to use a custom built LSM store instead: https://github.com/cockroachdb/pebble

Post reply on HN