How To Make An Infinitely Scalable RDBMS • Hacker News Board

Re: How To Make An Infinitely Scalable RDBMS

#31

Post by jbellis » Mon, Nov 25, 2013, 10:14 PM UTC

mtravis wrote: post #28

alanctgardner2 wrote: post #12

Earlier quoted context omitted.

CAP theorem can apply to any clustered system, it doesn't have to be multi-site. What happens if 6 of your 12 machines die? What if they get cut off from the other 6? edit: There's a bit of discussion further down about the SQL implementation. That's something I was very curious about as well. The projects linked below spend a lot of time working on supporting full ANSI SQL, and reducing latency by pushing down as ma…

Hi, Alan. Regarding CAP, I think that given redundant cluster interconnects, redundant managed power, odd # of cluster managers for quorum, all mean that split brain is just about out of the question, configured properly. The main reason that I have a FAQ about Hadoop is that I have been asked repeatedly by people, "what's the difference between InfiniSQL & Hadoop?" It seems to be the data project most on a lot of pe…

Ah, the myth of the sufficiently redundant network.

http://pl.atyp.us/wordpress/index.php/2010/10/when-partition...

http://kellabyte.com/2013/11/04/the-network-partitions-are-r...

Re: How To Make An Infinitely Scalable RDBMS

#32

Post by jbellis » Mon, Nov 25, 2013, 10:26 PM UTC

mtravis wrote: post #11

MichaelGG wrote: post #9

In-memory distributed database? VoltDB is already way past 500Ktx/sec on a 12-node cluster. On their site though, it says no sharding and that it can do these 500Ktx/sec even when each transaction involves data on multiple nodes. Does this performance degrade directly in relation to the number of nodes a tx needs to touch? A simple, straightforward, wire-level description of how things work when coordinating and perf…

Hi, Michael. Yes, VoltDB is very fast, but they self-admittedly do not perform well if transactions contain records spanning across multiple nodes. That is the key feature difference between InfiniSQL and VoltDB (along, of course, that their project is functionally much further along). If you want more details about how things work when performing transactions, I think that the overview I created would be a good star…

So basically, VoltDB acknowledges that cross-partition transactions are Hard and has put a lot of effort into minimizing them. (This is basically the entire point of the original HStore paper.)

InfiniSQL says don't worry, we'll just use 2PC. But not just yet, we're still working on the lock manager.

I look forward to your exegesis of how you plan to overcome the well-documented scaling problems with 2PC. Preferably after you have working code. :)

Re: How To Make An Infinitely Scalable RDBMS

#33

Post by nl » Mon, Nov 25, 2013, 10:46 PM UTC

JeffDClark wrote: post #19

glogla wrote: post #15

There was very interesting presentation by one professor. I'm not sure about what university, but he seemed to know his work. He talked about how databse world is about to change. ACID is really expensive in terms of resources, and so are the more difficult things about relational schema (foreign keys, checks, etc). And architecture of classic RDBMSes is pretty wasteful -- they use on-disk format but cache it in memo…

Is this the talk that you are referring to? http://slideshot.epfl.ch/play/suri_stonebraker

Note that Stonebraker makes some good points, but there are many ways to build scalability and Stonebraker is too fast to dismiss many.

In particular, his criticism of traditional databases seems based more on philosophy rather than evidence.

I'd advise reading both sides of the story:

http://lemire.me/blog/archives/2009/09/16/relational-databas...

http://lemire.me/blog/archives/2009/07/03/column-stores-and-...

http://architects.dzone.com/articles/stonebraker-talk-trigge...

http://gigaom.com/2011/07/11/amazons-werner-vogels-on-the-st...

http://dom.as/2011/07/08/stonebraker-trapped/

The date on some of those posts in interesting. 2009 is quite a while ago now, and I'd suggest that columnar datastores haven't exactly taken over. Some implementations have made some progress (eg Cassandra), but OTOH many non-traditional datastores have added traditional-database like features (eg, Facebook's SQL front end on their NoSQL system), and traditional databases have added NoSQL features too.

Re: How To Make An Infinitely Scalable RDBMS

#34

Post by mbesto » Mon, Nov 25, 2013, 10:47 PM UTC

glogla wrote: post #15

There was very interesting presentation by one professor. I'm not sure about what university, but he seemed to know his work. He talked about how databse world is about to change. ACID is really expensive in terms of resources, and so are the more difficult things about relational schema (foreign keys, checks, etc). And architecture of classic RDBMSes is pretty wasteful -- they use on-disk format but cache it in memo…

> But this seems to be first actual Main memory database.

Eh, not really...

This is exactly what SAP has been doing for several years via Hasso Plattner and the Potsdam Institute: https://epic.hpi.uni-potsdam.de/Home/HassoPlattner

If you've ever worked with large scale "enterprise" database warehouses, they tend to be slow and clunky. Back in 2006ish SAP took the whole Data Warehouse (well mainly just the data cubes) and chucked it into a columnar database (at the time it was called TREX, then became BW Accelerator) - http://en.wikipedia.org/wiki/TREX_search_engine

TREX exist way before 2006. SAP also bought a Korean company called P* (IIRC) which did non-columanr (traditional relational) and threw it into memory. SAP also had a produce called APO LiveCache - http://scn.sap.com/community/scm/apo/livecache - which lived around the same time.

This has now all evolved to a standard offering called SAP HANA - http://www.saphana.com/welcome - In it's second year of inception I believe SAP did roughly $360m in sales just on HANA alone.

Also, IIRC is InnoDB basically the open source version of exactly what you're talking about with "Postgres to main memory"?

edit- correction in TimesTen

Re: How To Make An Infinitely Scalable RDBMS

#35

Post by mtravis » Mon, Nov 25, 2013, 11:06 PM UTC

jbellis wrote: post #32

mtravis wrote: post #11

Earlier quoted context omitted.

Hi, Michael. Yes, VoltDB is very fast, but they self-admittedly do not perform well if transactions contain records spanning across multiple nodes. That is the key feature difference between InfiniSQL and VoltDB (along, of course, that their project is functionally much further along). If you want more details about how things work when performing transactions, I think that the overview I created would be a good star…

So basically, VoltDB acknowledges that cross-partition transactions are Hard and has put a lot of effort into minimizing them. (This is basically the entire point of the original HStore paper.) InfiniSQL says don't worry, we'll just use 2PC. But not just yet, we're still working on the lock manager. I look forward to your exegesis of how you plan to overcome the well-documented scaling problems with 2PC. Preferably a…

I'm not doing 2PC. I'm curious what gave you that impression, and I'll clarify any documents that seem to give off that impression.

I'd like InfiniSQL to handle hard workloads as well as it can, to minimize the amount that developers need to redesign their applications.

Please go to http://www.infinisql.org/community/ and follow the project on Twitter, and sign up for the newsletter, to keep apprised of progress.

Thanks for the input!

Re: How To Make An Infinitely Scalable RDBMS

#36

Post by mtravis » Mon, Nov 25, 2013, 11:12 PM UTC

brianberns wrote: post #14

An in-memory RDBMS hardly seems to be "infinitely scalable". How would this work with DBs in the terabyte size or larger?

Well, 2-way Cisco servers can hold 1TB RAM each.

It scales as long as throughput increases while new nodes are added. I've done benchmarking up to 12 nodes, and it continued to scale nearly linearly. (http://www.infinisql.org/blog). I'd like to push it further, but need $$$ for bigger benchmark environments.

Re: How To Make An Infinitely Scalable RDBMS

#37

Post by jbellis » Mon, Nov 25, 2013, 11:23 PM UTC

mtravis wrote: post #35

jbellis wrote: post #32

Earlier quoted context omitted.

So basically, VoltDB acknowledges that cross-partition transactions are Hard and has put a lot of effort into minimizing them. (This is basically the entire point of the original HStore paper.) InfiniSQL says don't worry, we'll just use 2PC. But not just yet, we're still working on the lock manager. I look forward to your exegesis of how you plan to overcome the well-documented scaling problems with 2PC. Preferably a…

I'm not doing 2PC. I'm curious what gave you that impression, and I'll clarify any documents that seem to give off that impression. I'd like InfiniSQL to handle hard workloads as well as it can, to minimize the amount that developers need to redesign their applications. Please go to http://www.infinisql.org/community/ and follow the project on Twitter, and sign up for the newsletter, to keep apprised of progress. Tha…

http://www.infinisql.org/docs/overview/#idp37097184 gave me that impression.

Re: How To Make An Infinitely Scalable RDBMS

#38

Post by MichaelGG » Mon, Nov 25, 2013, 11:24 PM UTC

mtravis wrote: post #11

MichaelGG wrote: post #9

In-memory distributed database? VoltDB is already way past 500Ktx/sec on a 12-node cluster. On their site though, it says no sharding and that it can do these 500Ktx/sec even when each transaction involves data on multiple nodes. Does this performance degrade directly in relation to the number of nodes a tx needs to touch? A simple, straightforward, wire-level description of how things work when coordinating and perf…

Hi, Michael. Yes, VoltDB is very fast, but they self-admittedly do not perform well if transactions contain records spanning across multiple nodes. That is the key feature difference between InfiniSQL and VoltDB (along, of course, that their project is functionally much further along). If you want more details about how things work when performing transactions, I think that the overview I created would be a good star…

Hey Mark. The overview seem to be much of the same. There's a lot of excited talk, which is fine, but should be limited to a leading paragraph. The fundamental issue is performance in face of transactions that need to do 2PC among multiple nodes (which also need to sync with the replicas).

I'm not much of an expert at all, but I like reading papers on databases. It seems to me that if you really did discover a breakthrough like this, you should be able to distill it to some basic algorithms and math. And a breakthrough of this scale would be quite notable.

If I'm reading correctly, there's no replica code even involved ATM. So 500Ktx/s really boils down to ~83Ktx/sec per node, on an in-memory database. Is it possible on modern hardware that this is just what to expect?

I am curious, and I'm not trying to be dismissive, but the copy sounds overly promising, without explaining how, even in theory, this will actually work. I'd suggest to explain that part first, then let the engineering come second.

Re: How To Make An Infinitely Scalable RDBMS

#39

Post by mtravis » Mon, Nov 25, 2013, 11:27 PM UTC

yid wrote: post #29

mtravis wrote: post #24

Earlier quoted context omitted.

Hi, yid. UPS protects against multiple simultaneous system crashes. Single system crash gets failed over, no problem. If both UPS systems detect their upstream PDU's as being out, then the InfiniSQL management protocol will initiate graceful shutdown, including persisting to disk. For write() issues, at least intially, I think that stuff in commodity hardware (such as ECC memory) is sufficient protection in most case…

I feel like you're banking a little too heavily on external measures for error protection/durability like UPSes and ECC memory, rather than embracing the inevitable fact that corruption and failures will occur. In short, I'd really need to see a white paper on why you're not reinventing the wheel before I'd use InifiSQL. Kudos for engaging the community though; please do keep us posted as you progress.

Actually, there's precious little that an application can do if a memory chip fails, or if ECC gets too many corrupted bits. If it gets too many corrupted bits, the kernel will generally do something like halt the system. I am not familiar with any application which does a write-read-write (or similar) to memory, but would be curious to learn about them. I'm sure such an algorithm can also be used in InfiniSQL.

I am familiar with applications such as IBM WebSphere MQ which does a triple-write to disk for every transaction, to overcome corruption problems. But even IBM will recommend turning that parameter off for performance reasons if the storage layer performs that kind of verification, such as HDS or EMC arrays. So, if an InfiniSQL user wants that level of storage protection, they can buy it.

Regarding InfiniSQL's planned use of UPS systems, that's pretty much the identical design to how the above-mentioned storage arrays protect block storage from power loss. I'm just moving the protection up to the application level.

Thanks for the conversation and thoughtful comments. Please go to http://www.infinisql.org/community/ and find the project on Twitter, sign up for newsletter, hit me on LinkedIn.

----

Hi, yid. For some reason I can't reply to your last comment. I think I didn't explain things well enough--I do plan to implement synchronous replication to protect against single replica failure. I describe the plan somewhat in http://www.infinisql.org/docs/overview/

Much of the code for the replication is in place (and it actually worked a few thousand lines of code ago) but it's currently not functional.

I have a big backlog to work on, but n-way synchronous replication is definitely in there.

Re: How To Make An Infinitely Scalable RDBMS

#40

Post by MichaelGG » Mon, Nov 25, 2013, 11:37 PM UTC

mtravis wrote: post #24

yid wrote: post #13

> UPS systems will stay active for a few minutes, based on their capacity, and the manager process will gracefuly shut down each daemon and write data to disk storage. This will ensure durability--even against power failure or system crash--while still maintaining in memory performance. How does a UPS ensure durability against system or program crashes, disk corruption in large clusters, and other failures that can a…

Hi, yid. UPS protects against multiple simultaneous system crashes. Single system crash gets failed over, no problem. If both UPS systems detect their upstream PDU's as being out, then the InfiniSQL management protocol will initiate graceful shutdown, including persisting to disk. For write() issues, at least intially, I think that stuff in commodity hardware (such as ECC memory) is sufficient protection in most case…

Well one example is that bug that caused a panic on a certain date (I think it may have been leap year related) - machines just die then. Only way to recover from that would be some neat reboot system that recovers the data from RAM. But, all systems have dataloss, so this problem affects any main-memory system.

You may get more volunteers by publishing a paper outlining the core concept. I clearly remember reading things like the H-Store paper, or the Dremel paper, and saying "damn, this makes sense and is really cool". Implementation details can be worked out and engineering approaches tried. But the underlying concept should be clear.