Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Databases at 14.4Mhz

blog.foundationdb.com

Post reply on HN
Print view View original

61–70 of 86 posts

Re: Databases at 14.4Mhz

#61

Post by grogers »

dchichkov wrote: post #55
imaginenore wrote: post #43

Earlier quoted context omitted.

But it doesn't store the data on disk. It should be compared to Memcached, not to a real storage engine.

Unlike Memcached or Redis, RAMCloud provides durability: * Automatic replication, crash recovery, and fail-over (no loss of availability if a server fails) * Durability guarantee: data is always replicated and durable before operations return, without significant performance hit (subject to the requirement for persistent buffers on backups).

You can't make a durable write to disk in 15us. SSDs are around 2 orders of magnitude slower than that. 15us is about right for in memory on two nodes in a single data center though. Most people wouldn't consider that durable though.

Re: Databases at 14.4Mhz

#62

Post by jedberg »

jrallison wrote: post #57
lclarkmichalek wrote: post #53

Earlier quoted context omitted.

Why do you need a 12 node cluster for 2 TB of data?

With a replication factor of 2 (for fault tolerance), it's ~4.5 TB. FoundationDB requires SSD drives, which the best we can get efficiently in our data center is ~670 GB of usable space (3x480GB raided). 12x670 = 8040 GB We try to keep extra space available for node failures (FDB will immediately start replicating addition data if it notices data with less than the configured number of replicas). Our dataset currentl…

Why do you use RAID on your nodes if you have an RF==2?

Re: Databases at 14.4Mhz

#63

Post by primitivesuave »

shortstuffsushi wrote: post #15

As someone who has no idea about the cost of high-scale computing like this, is $150/hr reasonable? It seems like an amount that's hard to sustain to me, but I have no idea if that's a steady, all the time rate, or a burst rate, or what. Or if it's a set up you'd actually ever even need -- seems like from the examples they mention (like the Tweets), they're above the need by a fair amount. Anyone else in this sort of…

There are literally thousands of companies that each hand over millions of dollars to Oracle on a semi-regular basis, for enormous server setups to store enormous data sets. This is a major improvement.

Re: Databases at 14.4Mhz

#64

Post by TomVDB »

anonymousDan wrote: post #54
Dave_Rosenthal wrote: post #35

Earlier quoted context omitted.

I think the difference is that FoundationDB has only one, and it's not an external simulation. The actual code that runs in production can also deterministically simulate a cluster of itself. I do believe this is unique among publically-available distributed databases.

Why is it a good thing that it's not an external simulation?

It makes everything completely determistic.

Re: Databases at 14.4Mhz

#65

Post by tinco »

jedberg wrote: post #62
jrallison wrote: post #57

Earlier quoted context omitted.

With a replication factor of 2 (for fault tolerance), it's ~4.5 TB. FoundationDB requires SSD drives, which the best we can get efficiently in our data center is ~670 GB of usable space (3x480GB raided). 12x670 = 8040 GB We try to keep extra space available for node failures (FDB will immediately start replicating addition data if it notices data with less than the configured number of replicas). Our dataset currentl…

Why do you use RAID on your nodes if you have an RF==2?

Perhaps to be able to recover without stressing other nodes? If a disk fails, your reads suddenly all go to 1 node in the replicaset. If then that same node also has to supply the data for the fresh harddrive, it might interfere with the read-performance and/or take a long time to restore full redundancy.

Also, in theory the raid 5 configuration would have faster reads.

Re: Databases at 14.4Mhz

#67

Post by jedberg »

tinco wrote: post #65
jedberg wrote: post #62

Earlier quoted context omitted.

Why do you use RAID on your nodes if you have an RF==2?

Perhaps to be able to recover without stressing other nodes? If a disk fails, your reads suddenly all go to 1 node in the replicaset. If then that same node also has to supply the data for the fresh harddrive, it might interfere with the read-performance and/or take a long time to restore full redundancy. Also, in theory the raid 5 configuration would have faster reads.

But then you might as well just use an RF of 3. You get all the benefits you listed above, plus more storage (5.7TB vs 4.5TB), and less configuration hassle. And greater horizontal scalability.

And a RAID 5 will never be faster than a RAID 0 or a JBOD. :)

Re: Databases at 14.4Mhz

#68

Post by jbergens »

vilda wrote: post #52
ChuckMcM wrote: post #20

This is Foundation DB's announcement they are doing full ACID databases with a 14.4M writes per second capability. That is insanely fast in the data base world. Running in AWS with 32 c3.8xlarge configured machines. So basically NSA level data base capability for $150/hr. But perhaps more interesting is that those same machines on the open market are about $225,000. That's two rack, one switch and a transaction rate…

Not all ACID transactions are equal. This is just a key-value store-like test. It shows the potential to scale, yet nothing regarding performance in real word. 32 c3.2xlarge instances have 1920GB memory. Given 1 billion 16B+ 8..100B values the whole dataset fits just into memory. The Cassandra test mentioned [1] sustained loss of 1/3 instances. That's very impressive! Would love to see how F-DB handles this type of r…

They do seem to talk about writes per second. That means you must flush to disk, not read from cache.

[Edit, more info] They seem to run with multiple clients which also stresses the system. From their explanation * The clients simulate 320,000 concurrent sessions

Re: Databases at 14.4Mhz

#69

Post by jrallison »

jedberg wrote: post #62
jrallison wrote: post #57

Earlier quoted context omitted.

With a replication factor of 2 (for fault tolerance), it's ~4.5 TB. FoundationDB requires SSD drives, which the best we can get efficiently in our data center is ~670 GB of usable space (3x480GB raided). 12x670 = 8040 GB We try to keep extra space available for node failures (FDB will immediately start replicating addition data if it notices data with less than the configured number of replicas). Our dataset currentl…

Why do you use RAID on your nodes if you have an RF==2?

A few reasons off the top of my head:

1) We're still interested in the nodes being as reliable as they can be. With RAID 5, we need two simultaneous disk failures to brick a node. With RAID 0 (to increase usable disk space), any of the 3 disks can brick the node.

Even with 12 nodes and RF of 2, an order of magnitude more node failures would be more likely to disrupt our service. Perhaps this makes more sense in a larger cluster with a higher RF factor?

2) We're using commodity hardware from a dedicated server provider, which uses RAID 1 or 5 configurations exclusively by default. We haven't reached the point where we felt that was enough of an issue or gain to investigate changing it.

3) Having more CPUs/disks in the cluster (12 nodes rather than cramming all data on 6) can be a good thing... as FDB scales fairly linearly.

Re: Databases at 14.4Mhz

#70

Post by itp »

anonymousDan wrote: post #54
Dave_Rosenthal wrote: post #35

Earlier quoted context omitted.

I think the difference is that FoundationDB has only one, and it's not an external simulation. The actual code that runs in production can also deterministically simulate a cluster of itself. I do believe this is unique among publically-available distributed databases.

Why is it a good thing that it's not an external simulation?

I'll give you some brief thoughts from my experience working at FoundationDB, but if you really want the in-depth answer to what makes our simulation framework an enabling technology for everything we do, you should take a look at the talk my colleague Will Wilson gave at Strange Loop [0].

Everything in our core is written in Flow, our asynchronous programming environment. The architecture of our server is essentially single threaded, with (pseudo-)concurrent actors allowing a single process to satisfy multiple roles at once.

The interfaces that these actors use to communicate -- with the disk, over the network, with the OS in general -- are abstracted and implemented in Flow as well. Each of these interfaces not only has at least one "real" implementation, but at least one, and sometimes more, simulated implementations.

Since we run multiple actors in a single process all the time, running yet more actors, pretending to be different machines, all still in the same process, was an obvious step. These workers communicate with each other via the _same_ network interfaces that real-world workers do in real clusters.

In the end we are able to become our own adversaries, pushing the limits of the system in ways that just don't happen often enough in the real world to test and debug in the wild. Our simulated implementations are allowed to present any behavior that can be observed in the real world, or justified by the spec, or implied by the contract. And they can do so much more frequently.

But most importantly -- as in, without this our system would never have been developed -- we can simulate these pathological behaviors in a completely deterministic fashion. Having the power to run a million tests, simulating multi-machine clusters trying to complete workloads while suffering from the most unbelievable combination of network partitions, dropped packets, failed disks, etc. etc., all while knowing that any error can be replayed, stepped through in a debugger, in a single process on a single machine... without this capability, we would never have had the confidence to build our product or evolve it as aggressively as we have.

tl;dr: We're not making simulations _of_ the system, we're running the system _in_ simulation, and that makes all the difference.

[0]: https://www.youtube.com/watch?v=4fFDFbi3toc

Post reply on HN