Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How To Make An Infinitely Scalable RDBMS

highscalability.com

Post reply on HN
Print view View original

41–50 of 92 posts

Re: How To Make An Infinitely Scalable RDBMS

#41

Post by yid »

mtravis wrote: post #39
yid wrote: post #29

Earlier quoted context omitted.

I feel like you're banking a little too heavily on external measures for error protection/durability like UPSes and ECC memory, rather than embracing the inevitable fact that corruption and failures will occur. In short, I'd really need to see a white paper on why you're not reinventing the wheel before I'd use InifiSQL. Kudos for engaging the community though; please do keep us posted as you progress.

Actually, there's precious little that an application can do if a memory chip fails, or if ECC gets too many corrupted bits. If it gets too many corrupted bits, the kernel will generally do something like halt the system. I am not familiar with any application which does a write-read-write (or similar) to memory, but would be curious to learn about them. I'm sure such an algorithm can also be used in InfiniSQL. I am…

> Actually, there's precious little that an application can do if a memory chip fails, or if ECC gets too many corrupted bits.

That's where replication and (distributed) consensus comes in, usually at the application level.

Re: How To Make An Infinitely Scalable RDBMS

#42

Post by mtravis »

jbellis wrote: post #31
mtravis wrote: post #28

Earlier quoted context omitted.

Hi, Alan. Regarding CAP, I think that given redundant cluster interconnects, redundant managed power, odd # of cluster managers for quorum, all mean that split brain is just about out of the question, configured properly. The main reason that I have a FAQ about Hadoop is that I have been asked repeatedly by people, "what's the difference between InfiniSQL & Hadoop?" It seems to be the data project most on a lot of pe…

Ah, the myth of the sufficiently redundant network. http://pl.atyp.us/wordpress/index.php/2010/10/when-partition... http://kellabyte.com/2013/11/04/the-network-partitions-are-r...

Redundant components and pathing, properly implemented and managed, are what allow enterprise storage arrays, mainframe clusters, Tandems, and the like, to operate 24x7 for years on end. Their myth seems to work.

Re: How To Make An Infinitely Scalable RDBMS

#43

Post by MichaelGG »

brianberns wrote: post #14

An in-memory RDBMS hardly seems to be "infinitely scalable". How would this work with DBs in the terabyte size or larger?

A terabyte of RAM is pretty cheap. Around $12K for the RAM. Last I quoted out a system for VoltDB, the total cost (complete servers with CPU, disk, RAM) came to ~$17/GB to $22/GB.

If you actually have transaction processing at this scale and need that performance, the RAM cost is not a major issue.

Re: How To Make An Infinitely Scalable RDBMS

#44

Post by mtravis »

MichaelGG wrote: post #40
mtravis wrote: post #24

Earlier quoted context omitted.

Hi, yid. UPS protects against multiple simultaneous system crashes. Single system crash gets failed over, no problem. If both UPS systems detect their upstream PDU's as being out, then the InfiniSQL management protocol will initiate graceful shutdown, including persisting to disk. For write() issues, at least intially, I think that stuff in commodity hardware (such as ECC memory) is sufficient protection in most case…

Well one example is that bug that caused a panic on a certain date (I think it may have been leap year related) - machines just die then. Only way to recover from that would be some neat reboot system that recovers the data from RAM. But, all systems have dataloss, so this problem affects any main-memory system. You may get more volunteers by publishing a paper outlining the core concept. I clearly remember reading t…

I'll consider this--meantime, will you please follow me on Twitter and/or sign up for the newsletter, so that you can be informed when the document is ready?

Re: How To Make An Infinitely Scalable RDBMS

#45

Post by vidarh »

mtravis wrote: post #28
alanctgardner2 wrote: post #12

Earlier quoted context omitted.

CAP theorem can apply to any clustered system, it doesn't have to be multi-site. What happens if 6 of your 12 machines die? What if they get cut off from the other 6? edit: There's a bit of discussion further down about the SQL implementation. That's something I was very curious about as well. The projects linked below spend a lot of time working on supporting full ANSI SQL, and reducing latency by pushing down as ma…

Hi, Alan. Regarding CAP, I think that given redundant cluster interconnects, redundant managed power, odd # of cluster managers for quorum, all mean that split brain is just about out of the question, configured properly. The main reason that I have a FAQ about Hadoop is that I have been asked repeatedly by people, "what's the difference between InfiniSQL & Hadoop?" It seems to be the data project most on a lot of pe…

> configured properly.

... and there is where it falls apart. Sooner or later, "someone" is going to do something Incredibly Dumb that is going to take down a lot of nodes.

If you are betting that you can just add enough redundancy split brain can't happen, I have to question why I should take you seriously with important data.

(It also indicates to me this is going to be ludicrously expensive to set up though, but that's another issue)

Re: How To Make An Infinitely Scalable RDBMS

#46

Post by gnaritas »

mtravis wrote: post #28
alanctgardner2 wrote: post #12

Earlier quoted context omitted.

CAP theorem can apply to any clustered system, it doesn't have to be multi-site. What happens if 6 of your 12 machines die? What if they get cut off from the other 6? edit: There's a bit of discussion further down about the SQL implementation. That's something I was very curious about as well. The projects linked below spend a lot of time working on supporting full ANSI SQL, and reducing latency by pushing down as ma…

Hi, Alan. Regarding CAP, I think that given redundant cluster interconnects, redundant managed power, odd # of cluster managers for quorum, all mean that split brain is just about out of the question, configured properly. The main reason that I have a FAQ about Hadoop is that I have been asked repeatedly by people, "what's the difference between InfiniSQL & Hadoop?" It seems to be the data project most on a lot of pe…

> Regarding CAP, I think that given redundant cluster interconnects, redundant managed power, odd # of cluster managers for quorum, all mean that split brain is just about out of the question, configured properly.

Failure is never out of the question, you either plan for it or you suffer from it. CAP applies.

Re: How To Make An Infinitely Scalable RDBMS

#47

Post by mtravis »

jbellis wrote: post #37
mtravis wrote: post #35

Earlier quoted context omitted.

I'm not doing 2PC. I'm curious what gave you that impression, and I'll clarify any documents that seem to give off that impression. I'd like InfiniSQL to handle hard workloads as well as it can, to minimize the amount that developers need to redesign their applications. Please go to http://www.infinisql.org/community/ and follow the project on Twitter, and sign up for the newsletter, to keep apprised of progress. Tha…

http://www.infinisql.org/docs/overview/#idp37097184 gave me that impression.

Oh, I thought you said 2PC (two-phased commit).

Yes, InfiniSQL uses two phase locking. The Achilles' Heel is deadlock management. By necessity, the deadlock management will need to be single-threaded (at least as far as I can figure). No arguments there, so deadlock-prone usage patterns may definitely be problematic.

I don't think there's a perfect concurrency management protocol. MVCC is limited by transactionid generation. Predictive locking can't get rolled back once it starts, or limits practical throughput to single partitions.

2PL works. It's not ground-breaking (though incorporating it in the context of inter-thread messaging might be unique). And it will scale fine other than for a type of workload that tends to bring about a lot of deadlocks.

Re: How To Make An Infinitely Scalable RDBMS

#48

Post by vidarh »

mtravis wrote: post #42
jbellis wrote: post #31

Earlier quoted context omitted.

Ah, the myth of the sufficiently redundant network. http://pl.atyp.us/wordpress/index.php/2010/10/when-partition... http://kellabyte.com/2013/11/04/the-network-partitions-are-r...

Redundant components and pathing, properly implemented and managed, are what allow enterprise storage arrays, mainframe clusters, Tandems, and the like, to operate 24x7 for years on end. Their myth seems to work.

Yes, but not without addressing what should happen when split brain occurs.

Re: How To Make An Infinitely Scalable RDBMS

#50

Post by mtravis »

vidarh wrote: post #45
mtravis wrote: post #28

Earlier quoted context omitted.

Hi, Alan. Regarding CAP, I think that given redundant cluster interconnects, redundant managed power, odd # of cluster managers for quorum, all mean that split brain is just about out of the question, configured properly. The main reason that I have a FAQ about Hadoop is that I have been asked repeatedly by people, "what's the difference between InfiniSQL & Hadoop?" It seems to be the data project most on a lot of pe…

> configured properly. ... and there is where it falls apart. Sooner or later, "someone" is going to do something Incredibly Dumb that is going to take down a lot of nodes. If you are betting that you can just add enough redundancy split brain can't happen, I have to question why I should take you seriously with important data. (It also indicates to me this is going to be ludicrously expensive to set up though, but t…

Redundancy, quorum protocol, proactive testing, rigorous QA.

And actual 24x7 environments with important data are ludicrously expensive. I expect InfiniSQL to be less expensive since it's based on x86_64, Linux, is open source. But yeah, hardware and environment need to be right.

I'm not sure what I said wrong, but I'm nowhere claiming that split brain (or any failure scenario) can be ruled out entirely in all circumstances--but in practice, split brain is avoided 24x7 for years on end in many different architectures.

It's not magic.

Just like you can't have "enough" storage redundancy. You can have a 100-way mirror of hard drives that will still lose data if you lose 100 disks in less time than somebody replaces one of them.

Post reply on HN