Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Distributed SQLite for Go applications

github.com

Post reply on HN
Print view View original

41–50 of 69 posts

Re: Distributed SQLite for Go applications

#41

Post by zzzcpan »

atombender wrote: post #16

It's fascinating how Raft has democratized distributed computing -- writing a consistent, distributed state machine now can be done in a few lines of code, assuming you have a Raft implementation lying around, like Hashicorp's excellent Go library. What this project (and similar projects such as Rqlite) doesn't address is the hard problem -- sharding. Raft makes it quite trivial to write a master/slave system where e…

> writing a consistent, distributed state machine now can be done in a few lines of code

Well, if you run it on a very fast and reliable local network without much load, not multi dc deployments over public internet, than sure, it can sort of work. Not without problems though once a fault occurs.

Re: Distributed SQLite for Go applications

#42

Post by otoolep »

atombender wrote: post #16

It's fascinating how Raft has democratized distributed computing -- writing a consistent, distributed state machine now can be done in a few lines of code, assuming you have a Raft implementation lying around, like Hashicorp's excellent Go library. What this project (and similar projects such as Rqlite) doesn't address is the hard problem -- sharding. Raft makes it quite trivial to write a master/slave system where e…

rqlite author here. I am considering adding distributed transaction support to rqlite (it already has a form of transaction support), if I can come up with a solid implementation, while keeping rqlite simple to deploy and operate (which is a primary goal of rqlite):

https://groups.google.com/forum/#!topic/raft-dev/BaS5Z2NkDxA

https://github.com/rqlite/rqlite/issues/266

dqlite is very interesting. If its patches to SQLite are accepted upstream, dqlite could become the storage engine for rqlite. But right now rqlite is deliberately built on vanilla-SQLite, so it can offer the same correctness guarantees as SQLite.

Re: Distributed SQLite for Go applications

#43

Post by otoolep »

zzzcpan wrote: post #41
atombender wrote: post #16

It's fascinating how Raft has democratized distributed computing -- writing a consistent, distributed state machine now can be done in a few lines of code, assuming you have a Raft implementation lying around, like Hashicorp's excellent Go library. What this project (and similar projects such as Rqlite) doesn't address is the hard problem -- sharding. Raft makes it quite trivial to write a master/slave system where e…

> writing a consistent, distributed state machine now can be done in a few lines of code Well, if you run it on a very fast and reliable local network without much load, not multi dc deployments over public internet, than sure, it can sort of work. Not without problems though once a fault occurs.

I don't agree. The whole point of putting a solid Raft implementation at the center of your system is deal with the faults. I myself built a simple distributed state machine which deals with faults perfectly well. Of course I built on top of a good Raft implementation, which certainly is not a few lines of code.

https://github.com/otoolep/hraftd

Re: Distributed SQLite for Go applications

#45

Post by otoolep »

hardwaresofton wrote: post #3

Damn it, I was going to work on exactly this -- my goals were slightly different but basically the same paxos/raft/gossip + sqlite and see how far I could take it. Note for those who are interested in cool-stuff-with-SQLite, there's a similar project called RQLite (mentioned in the README): https://github.com/rqlite/rqlite The main differences are laid out in the README, but IMO it really all boils down to the fact t…

rqlite is also a full application, whereas dqlite is more like the SQLite library -- you must wrap an application layer around it.

In principle rqlite could use dqlite.

Re: Distributed SQLite for Go applications

#46

Post by usermac »

jxub wrote: post #37
atombender wrote: post #16

It's fascinating how Raft has democratized distributed computing -- writing a consistent, distributed state machine now can be done in a few lines of code, assuming you have a Raft implementation lying around, like Hashicorp's excellent Go library. What this project (and similar projects such as Rqlite) doesn't address is the hard problem -- sharding. Raft makes it quite trivial to write a master/slave system where e…

An excellent interactive explanation of Raft can be found here: http://thesecretlivesofdata.com/raft/ . It would be interesting as hell to build a small implementation in the language of choice (elixir?, pony?).

That was good to see. Thank you.

Re: Distributed SQLite for Go applications

#47

Post by otoolep »

atombender wrote: post #16

It's fascinating how Raft has democratized distributed computing -- writing a consistent, distributed state machine now can be done in a few lines of code, assuming you have a Raft implementation lying around, like Hashicorp's excellent Go library. What this project (and similar projects such as Rqlite) doesn't address is the hard problem -- sharding. Raft makes it quite trivial to write a master/slave system where e…

> What this project (and similar projects such as rqlite) doesn't address is the hard problem -- sharding.

Yes, this is technically correct. However to meet its goals rqlite (and dqlites) does not require sharding. The point of rqlite is to provide fault-tolerance and high-availability. Again, you are technically correct that rqlite doesn't support sharding, but sharding simply isn't required for rqlite to do what it wants to do. Sharding is a solution to a different type of problem.

Re: Distributed SQLite for Go applications

#48

Post by shalabhc »

atombender wrote: post #16

It's fascinating how Raft has democratized distributed computing -- writing a consistent, distributed state machine now can be done in a few lines of code, assuming you have a Raft implementation lying around, like Hashicorp's excellent Go library. What this project (and similar projects such as Rqlite) doesn't address is the hard problem -- sharding. Raft makes it quite trivial to write a master/slave system where e…

Check out http://www.actordb.com/ for a massively sharded sqlite system.

Re: Distributed SQLite for Go applications

#49

Post by zzzcpan »

otoolep wrote: post #43
zzzcpan wrote: post #41

Earlier quoted context omitted.

> writing a consistent, distributed state machine now can be done in a few lines of code Well, if you run it on a very fast and reliable local network without much load, not multi dc deployments over public internet, than sure, it can sort of work. Not without problems though once a fault occurs.

I don't agree. The whole point of putting a solid Raft implementation at the center of your system is deal with the faults. I myself built a simple distributed state machine which deals with faults perfectly well. Of course I built on top of a good Raft implementation, which certainly is not a few lines of code. https://github.com/otoolep/hraftd

No, Raft doesn't deal with the faults, it deals with consensus and can help tolerate certain faults. You then deal with the faults by other means. And even on a fast and reliable local network where Raft can work well, once you have a significant amount of data replacing failing nodes, healing broken records, resyncing, rebalancing all without affecting operations is not trivial and unlikely to be done properly.

Re: Distributed SQLite for Go applications

#50

Post by otoolep »

zzzcpan wrote: post #49
otoolep wrote: post #43

Earlier quoted context omitted.

I don't agree. The whole point of putting a solid Raft implementation at the center of your system is deal with the faults. I myself built a simple distributed state machine which deals with faults perfectly well. Of course I built on top of a good Raft implementation, which certainly is not a few lines of code. https://github.com/otoolep/hraftd

No, Raft doesn't deal with the faults, it deals with consensus and can help tolerate certain faults. You then deal with the faults by other means. And even on a fast and reliable local network where Raft can work well, once you have a significant amount of data replacing failing nodes, healing broken records, resyncing, rebalancing all without affecting operations is not trivial and unlikely to be done properly.

> Raft doesn't deal with the faults

I'm not sure what this means. The Raft paper explicitly states that Raft is a fault-tolerant system -- and by definition this means it deals with faults. To quote the paper:

"Replicated state machines are used to solve a variety of fault tolerance problems in distributed systems."

Raft is a type of replicated state machine. To say that "Raft doesn't deal with faults" is not correct. Perhaps you mean that there is other work to be done to take the fault tolerance offered by Raft and build a functioning application, and a system that will stay up in the real world -- and deal with an even wider range of faults. That I definitely agree with.

Post reply on HN