Distributed SQLite for Go applications • Hacker News Board

Re: Distributed SQLite for Go applications

#21

Post by free-ekanayaka » Fri, Apr 06, 2018, 6:31 AM UTC

PirateBay wrote: post #5

Projects like this(sparsely documented, tiny) always seem to be just internal tools that the company just threw out into the world just because they can. Not that I'm ungrateful, but without any sort of explanation as to why I'd use this over a clustered MySQL or Postgres it's a little hard to make heads or tails about whether I should care.

I'm working towards a 1.0 release. At that point documentation will be more complete and the plumbing needed to use it in an app should be also reduced.

I think the best explanation of why you'd use it over PG or MySQL is the same as why you'd use SQLite over them:

https://www.sqlite.org/whentouse.html

With dqlite you also get HA, fault-tolerance and transparent failover, if you're willing to trade it with a bit of performance.

Re: Distributed SQLite for Go applications

#22

Post by contingencies » Fri, Apr 06, 2018, 6:40 AM UTC

strken wrote: post #9

PirateBay wrote: post #5

Projects like this(sparsely documented, tiny) always seem to be just internal tools that the company just threw out into the world just because they can. Not that I'm ungrateful, but without any sort of explanation as to why I'd use this over a clustered MySQL or Postgres it's a little hard to make heads or tails about whether I should care.

It looks like a dependency of lxd, the successor to lxc, which is being developed by Canonical (of Ubuntu fame). You can see how they're using it in https://github.com/lxc/lxd/tree/master/lxd/cluster . Dqlite and rqlite are not in the same space as postgres and mysql, they're more comparable to tools like Zookeeper, etcd, or Consul[0] - low throughput CP data stores that are generally used for coordinating distribute…

Just what the world needs, yet another would-you-like-my-version-of-clustering-with-that virtual environments tool...

Re: Distributed SQLite for Go applications

#23

Post by eloff » Fri, Apr 06, 2018, 6:56 AM UTC

PirateBay wrote: post #5

Projects like this(sparsely documented, tiny) always seem to be just internal tools that the company just threw out into the world just because they can. Not that I'm ungrateful, but without any sort of explanation as to why I'd use this over a clustered MySQL or Postgres it's a little hard to make heads or tails about whether I should care.

It's work someone else did and shared with you for free. It strikes me as lazy to complain about how you don't know what to do with it and want everything spoon-fed to you. It's go+sqlite+raft. That means it's an embedded database for Go that can be used across a cluster of Go processes while maintaining ACID goodness. I got that out of it in 2 minutes of looking at the readme, I think you could manage the same if you take the time it took you here to make a new account and comment.

Re: Distributed SQLite for Go applications

#24

Post by notamy » Fri, Apr 06, 2018, 7:02 AM UTC

free-ekanayaka wrote: post #18

notamy wrote: post #2

After looking through the repo, I'm still not sure exactly why you would want / need this. What exactly is the use-case here?

If your application is: 1) Distributed (e.g. n equal processes running on n machines) 2) Needs some shared state across nodes 3) Would like that state to have SQL semantics (relations, transactions, etc.) 4) Not to heavy on writes to this shared state (I might provide some ballpark numbers at some point) 5) Wants to avoid the operational overhead of an external storage system (mysql, postgresql) 6) Wants to be fault-…

Interesting. Thank you for the explanation!

Re: Distributed SQLite for Go applications

#25

Post by nudpiedo » Fri, Apr 06, 2018, 8:13 AM UTC

I wonder how this kind of project can be tested in order to ensure the correctness of the use cases and error cases.

does anyone know a resource to learn more about testing in this kind of situation? I hope there is an alternative to just try all cases in different real hardware setups (which is the mantra at my company)

Re: Distributed SQLite for Go applications

#26

Post by lifty » Fri, Apr 06, 2018, 9:17 AM UTC

nudpiedo wrote: post #25

I wonder how this kind of project can be tested in order to ensure the correctness of the use cases and error cases. does anyone know a resource to learn more about testing in this kind of situation? I hope there is an alternative to just try all cases in different real hardware setups (which is the mantra at my company)

One very good resource are the writings of Aphyr where he tests various distributed databases: https://aphyr.com/tags/jepsen . Jepsen is the framework he wrote to test them. Next would be formal verification but you get very far by understanding the distributed db semantics and applying practical tests.

Re: Distributed SQLite for Go applications

#27

Post by eis » Fri, Apr 06, 2018, 9:21 AM UTC

atombender wrote: post #16

It's fascinating how Raft has democratized distributed computing -- writing a consistent, distributed state machine now can be done in a few lines of code, assuming you have a Raft implementation lying around, like Hashicorp's excellent Go library. What this project (and similar projects such as Rqlite) doesn't address is the hard problem -- sharding. Raft makes it quite trivial to write a master/slave system where e…

  > It's fascinating how Raft has democratized distributed computing

First I thought you probably ment commoditized but then I realized how democratized actually applies perfectly well to consensus algorithms because by definition they apply decisions via voting by all participants :)

But anyways, I agree very much with you that Raft has transformed the industry and we can't be thankful enough for it. But like you said, it's just one of the core building blocks of a distributed system that also shards data. Sharding is more use-case specific and so it can be valid for different databases to have different sharding logics.

Re: Distributed SQLite for Go applications

#28

Post by free-ekanayaka » Fri, Apr 06, 2018, 9:25 AM UTC

atombender wrote: post #16

It's fascinating how Raft has democratized distributed computing -- writing a consistent, distributed state machine now can be done in a few lines of code, assuming you have a Raft implementation lying around, like Hashicorp's excellent Go library. What this project (and similar projects such as Rqlite) doesn't address is the hard problem -- sharding. Raft makes it quite trivial to write a master/slave system where e…

If you require SQL, sharding and concurrent writes, yes do use CockroachDB or other equivalent solution.

If you don't have those requirements, dqlite or rqlite might get the job done with less moving parts and operational overhead.

It's pretty much the same argument of using SQLite vs (say) PostgreSQL, but translated into distributed systems. See:

https://www.sqlite.org/whentouse.html

Re: Distributed SQLite for Go applications

#29

Post by majewsky » Fri, Apr 06, 2018, 9:31 AM UTC

bbayer wrote: post #12

sqlite is fantastic piece of software. I am using it in every single side projects of mine. I can not describe how I satisfied with results even in relatively high load (300K pages/day on $5 Linode VPS). I found that most of time using a full featured RDBMS is not necessary depending on use case.(I am doing mostly reads)

> 300K pages/day

Does that qualify as "high load"? That's just below 4 pages/second. Given how beefy server CPUs and SSDs are, I would consider something like 100+ requests/second "high load".

Re: Distributed SQLite for Go applications

#30

Post by free-ekanayaka » Fri, Apr 06, 2018, 9:31 AM UTC

nudpiedo wrote: post #25

I wonder how this kind of project can be tested in order to ensure the correctness of the use cases and error cases. does anyone know a resource to learn more about testing in this kind of situation? I hope there is an alternative to just try all cases in different real hardware setups (which is the mantra at my company)

Unit and fuzzy-based test coverage that includes all possible edge cases is being put in place. I'd like to write a Jepsen test suite too.

Hardware does not matter that much, since SQLite is pretty hardened for handling any possible hardware failure (out of memory, out of disk space, memory/disk corruption) and propagate that to client code (including dqlite). Regarding network failures (hardware-related or not), raft guards you against them, and it's formally proved.