Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5K Hardware

highscalability.com

Post reply on HN
Print view View original

31–40 of 45 posts

Re: Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5K Hardware

#31

Post by leif »

dschiptsov wrote: post #29

T in TPS means Transactions, right? What kind of a transaction is that one which uses no persistent storage?) 1 million network-to-memory writes, well, that is quite possible, but, please, do not call this a transaction in the way it meant in TPS.) What was meant in the old days by transaction, was an atomic operation which completes after storing the data in a persistent (usually direct-access, which means no buffer…

You can achieve durability with very high performance using write-ahead logging, lots of concurrent writers doing group commit for the log fsyncs, and a write-optimized data structure like an LSM tree or a fractal tree. Maybe not 1 million on a $5k server, but you can get a lot closer than I imagine you're picturing right now.

In any case what they seem to be measuring is a read-only, in memory workload, so this is not that impressive. IIRC, InnoDB has no trouble pulling off something like this.

Re: Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5K Hardware

#32

Post by dschiptsov »

leif wrote: post #31
dschiptsov wrote: post #29

T in TPS means Transactions, right? What kind of a transaction is that one which uses no persistent storage?) 1 million network-to-memory writes, well, that is quite possible, but, please, do not call this a transaction in the way it meant in TPS.) What was meant in the old days by transaction, was an atomic operation which completes after storing the data in a persistent (usually direct-access, which means no buffer…

You can achieve durability with very high performance using write-ahead logging, lots of concurrent writers doing group commit for the log fsyncs, and a write-optimized data structure like an LSM tree or a fractal tree. Maybe not 1 million on a $5k server, but you can get a lot closer than I imagine you're picturing right now. In any case what they seem to be measuring is a read-only, in memory workload, so this is n…

So, disk writes are necessary, after all?

Yes, there are lots of tricks, like placing that append-only physical transaction log on a different controller with a distinct storage device, etc. Data partitioning is the another big idea. Having indexes in memory to avoid unnecessary reads, using collected statistics in a query optimizer, etc. But nothing could beat the partitioning based on actual workloads and separation of tablespaces on distinct hardware, including decoupling indexes from the tables - this is what DBAs were for.

I used to be Informix DBA in old good days, so I can't help but smile when I look at MySQL (well, they added lots of partitioning options in recent InnoDB - the things Informix could do out of box 12 years ago) leave alone modern NoFsync "databases".)

Btw, not all NoSQL guys are insane.) Riak with LevelDB storage backend is very sane approach, which cares about and counts writes.

Re: Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5K Hardware

#33

Post by leif »

dschiptsov wrote: post #32
leif wrote: post #31

Earlier quoted context omitted.

You can achieve durability with very high performance using write-ahead logging, lots of concurrent writers doing group commit for the log fsyncs, and a write-optimized data structure like an LSM tree or a fractal tree. Maybe not 1 million on a $5k server, but you can get a lot closer than I imagine you're picturing right now. In any case what they seem to be measuring is a read-only, in memory workload, so this is n…

So, disk writes are necessary, after all? Yes, there are lots of tricks, like placing that append-only physical transaction log on a different controller with a distinct storage device, etc. Data partitioning is the another big idea. Having indexes in memory to avoid unnecessary reads, using collected statistics in a query optimizer, etc. But nothing could beat the partitioning based on actual workloads and separatio…

Of course they are, the trick is to get the most utility out of each one. I work at Tokutek where we use a data structure that does this, in the sense that when your working set is larger than RAM, we still don't incur very many I/Os for writes. If you want durability, there's nothing you can do about the logging fsyncs except buy yourself a nice battery-backed disk controller.

Partitioning is often a bad idea because it messes with your queries. I don't know what's new about partitioning in InnoDB but I think it's generally a symptom of the over-use of B-trees, which don't try to do anything smart about random writes. The change buffer is a decent idea but it's just a stopgap, when you have enough data it doesn't make a dent any more. A better idea is to use a data structure that can handle lots of writes.

I have just started learning about Riak, and from what I understand, they need to do a query (so, a disk seek) on every insert (to calculate something with vector clocks), so they aren't actually using the write optimization that LevelDB's LSM-trees can provide. I don't actually think it should provide that fantastic performance, but I should admit I haven't run it yet. Maybe they're more interested in the compression LevelDB gives them.

Shameless plug time! http://www.tokutek.com/2011/09/write-optimization-myths-comp...

Re: Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5K Hardware

#35

Post by jamesaguilar »

bradhe wrote: post #34

Somewhat off topic, but this article perfectly shows how difficult taking full advantage of a multi-core environment truly is, and how lacking a lot of our tools are. Granted, this is an extreme example.

Our tools for something like this aren't great because almost nobody needs to do 1M reads per second, and of those who do, only a tiny fraction have a working set that can fit economically in memory and is read-only, like the system in this article.

Re: Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5K Hardware

#36

Post by tracker1 »

dschiptsov wrote: post #32
leif wrote: post #31

Earlier quoted context omitted.

You can achieve durability with very high performance using write-ahead logging, lots of concurrent writers doing group commit for the log fsyncs, and a write-optimized data structure like an LSM tree or a fractal tree. Maybe not 1 million on a $5k server, but you can get a lot closer than I imagine you're picturing right now. In any case what they seem to be measuring is a read-only, in memory workload, so this is n…

So, disk writes are necessary, after all? Yes, there are lots of tricks, like placing that append-only physical transaction log on a different controller with a distinct storage device, etc. Data partitioning is the another big idea. Having indexes in memory to avoid unnecessary reads, using collected statistics in a query optimizer, etc. But nothing could beat the partitioning based on actual workloads and separatio…

There's also RethinkDB which seems to be focused on the D in acid while being a non-relational database. When you really need performance, in general relationships/joins need to go out the window as much as possible, and often one or more of the letters in ACID are compromised.

It should get very interesting in the next couple of years.. of course MOST environments don't need the kind of performance or scale that these systems are really offering.

IIRC StackOverflow ran for a very long time on a single server, under some pretty serious demand. In some cases SQL with a caching system for mostly-read data can be better... other scenarios tend to fit a document (non-relational) data store better.. just depends.

Re: Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5K Hardware

#37

Post by tracker1 »

rorrr2 wrote: post #21

How often OS/network/context overhead is the bottleneck? In my experience most of the time it's the DB. Even if it fits in RAM, complex queries always take most of the time. (Web dev here).

That's kind of a key argument in NoSQL data stores... In many scenarios having a key/value store is better, a single key to lookup a record and all the necessary related data.

Some hybrid stores like MongoDB, RethinkDB and others offer more characteristics similar a traditional SQL RDBMS, while offering horizontal scaling. It's when you have several join operations that performance really takes a hit under significant load. You can't really scale a relational database in the same way.

That said, as much as I like non-relational databases, they aren't the best fit for every use case. Beyond this, most situations don't need that type of performance scaling.

Re: Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5K Hardware

#38

Post by ignoranimus »

Gee, that's spectacular. Now...

If only they would do the reader the simple, and most gracious service of defining precisely what they mean by this obscure "TPS" acronym.

...and before you downvote this comment (because I can smell your itchy little fingers all the way from the otherside of the internet), yes, I can assure you that I did actually Google for the answer. And yes, I did discern what is meant by TPS.

But that isn't the point. The point isn't that I'm a lazy slacker, and/or an ignorant yokle because I didn't already know the meaning of the abreviation innately, and feel inconvenienced by having to open another browser window, and search for some clue.

The point is that the author is assuming everyone will immediately know and understand that acronym, but meanwhile, when I conduct my search, I am forced to assume that my chosen definition is correct, wihout actually knowing for sure.

And for that reason, I'm going to leave out the meaning I've chosen as the author's intended definition for TPS. I have no way of knowing whether my assumption was accurate. So the mystery persists. What does TPS actually mean? Go search for it, you lazy, ignorant slacker.

Re: Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5K Hardware

#39

Post by kintamanimatt »

ignoranimus wrote: post #38

Gee, that's spectacular. Now... If only they would do the reader the simple, and most gracious service of defining precisely what they mean by this obscure "TPS" acronym. ...and before you downvote this comment (because I can smell your itchy little fingers all the way from the otherside of the internet), yes, I can assure you that I did actually Google for the answer. And yes, I did discern what is meant by TPS. But…

There's little excuse for laziness when pursuing new knowledge. A two second Google search and a new tab is not a burden. If you didn't get the answer first time, you should qualify the search with one of the other keywords from the article.

A certain level of reader sophistication is assumed, and not everything is explained with a high level of hand-holding. It's a technical tutorial, not a beginner's tutorial.

TPS = transactions per second.

Re: Russ’ 10 Ingredient Recipe for Making 1 Million TPS on $5K Hardware

#40

Post by regularfry »

kintamanimatt wrote: post #39
ignoranimus wrote: post #38

Gee, that's spectacular. Now... If only they would do the reader the simple, and most gracious service of defining precisely what they mean by this obscure "TPS" acronym. ...and before you downvote this comment (because I can smell your itchy little fingers all the way from the otherside of the internet), yes, I can assure you that I did actually Google for the answer. And yes, I did discern what is meant by TPS. But…

There's little excuse for laziness when pursuing new knowledge. A two second Google search and a new tab is not a burden. If you didn't get the answer first time, you should qualify the search with one of the other keywords from the article. A certain level of reader sophistication is assumed, and not everything is explained with a high level of hand-holding. It's a technical tutorial, not a beginner's tutorial. TPS…

In general you'd be right, but in this specific case there's a fair bit of ambiguity as to what "transaction" is taken to mean. A definition or two wouldn't have gone amiss here.
Post reply on HN