Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Distributed SQLite for Go applications

github.com

Post reply on HN
Print view View original

11–20 of 69 posts

Re: Distributed SQLite for Go applications

#11

Post by bluejekyll »

hardwaresofton wrote: post #8
bluejekyll wrote: post #7

Earlier quoted context omitted.

Just because somethings already been done, doesn’t mean you can do it too, or even better.

Yeah, but it's probably not smart to re-invent the wheel too many times. Especially when the projects that came before don't look badly written or unreasonable. I think I do have something to offer -- mostly trying gossip and making use of aggressive automatic sharding -- but I'm also more interested in what I wanted to build on top of distributed SQLite as well. No need to get down in the weeds if someone's already…

All great points. I was just responding to what seemed like a defeated perspective.

By all means contribute to this project. Everything starts somewhere!

Re: Distributed SQLite for Go applications

#12

Post by bbayer »

sqlite is fantastic piece of software. I am using it in every single side projects of mine. I can not describe how I satisfied with results even in relatively high load (300K pages/day on $5 Linode VPS). I found that most of time using a full featured RDBMS is not necessary depending on use case.(I am doing mostly reads)

Re: Distributed SQLite for Go applications

#13

Post by xenadu02 »

hardwaresofton wrote: post #3

Damn it, I was going to work on exactly this -- my goals were slightly different but basically the same paxos/raft/gossip + sqlite and see how far I could take it. Note for those who are interested in cool-stuff-with-SQLite, there's a similar project called RQLite (mentioned in the README): https://github.com/rqlite/rqlite The main differences are laid out in the README, but IMO it really all boils down to the fact t…

The WAL frames are by definition the result of the executed transactions so replicating the frames is a pretty smart way of ensuring you correctly distribute the results.

You're correct: only the master accepts writes. During a failover clients can get an error and are expected to retry against the new master.

Re: Distributed SQLite for Go applications

#14

Post by hardwaresofton »

xenadu02 wrote: post #13
hardwaresofton wrote: post #3

Damn it, I was going to work on exactly this -- my goals were slightly different but basically the same paxos/raft/gossip + sqlite and see how far I could take it. Note for those who are interested in cool-stuff-with-SQLite, there's a similar project called RQLite (mentioned in the README): https://github.com/rqlite/rqlite The main differences are laid out in the README, but IMO it really all boils down to the fact t…

The WAL frames are by definition the result of the executed transactions so replicating the frames is a pretty smart way of ensuring you correctly distribute the results. You're correct: only the master accepts writes. During a failover clients can get an error and are expected to retry against the new master.

Oh this is what I figured it was, it's the only thing left! I just have never come acros the teminology of a WAL "frame" before, so I wasn't completley sure. since it's a log I would have expected it to be called a record.

Loud and clear on the failure modes, the README was pretty clear about it

Re: Distributed SQLite for Go applications

#15

Post by free-ekanayaka »

hardwaresofton wrote: post #3

Damn it, I was going to work on exactly this -- my goals were slightly different but basically the same paxos/raft/gossip + sqlite and see how far I could take it. Note for those who are interested in cool-stuff-with-SQLite, there's a similar project called RQLite (mentioned in the README): https://github.com/rqlite/rqlite The main differences are laid out in the README, but IMO it really all boils down to the fact t…

Hello, author of dqlite here.

Things are still in flux, but yes, I plan to publish benchmarks before making a 1.0 release, as well as improving documentations and introduce some more abstractions to make it easier to use.

The reason it might be slower is some cases is that RQlite replicates statements and dqlite replicates WAL frames (where "frames" roughly means "disk pages"), which are typically bigger.

I didn't run benchmarks against RQlite yet, but I'd expect performance differences to be negligible for most use cases.

Re: Distributed SQLite for Go applications

#16

Post by atombender »

It's fascinating how Raft has democratized distributed computing -- writing a consistent, distributed state machine now can be done in a few lines of code, assuming you have a Raft implementation lying around, like Hashicorp's excellent Go library.

What this project (and similar projects such as Rqlite) doesn't address is the hard problem -- sharding. Raft makes it quite trivial to write a master/slave system where every node is guaranteed to be identical, but it won't scale writes and won't distribute storage across your cluster.

You can build sharding on top of Raft, but in an SQL setting this would require distributed transactions, something that's difficult enough that relatively few projects have tackled it so far, the notable ones being CockroachDB and TiDB. (Google Spanner, being proprietary, is not relevant in this context.)

Re: Distributed SQLite for Go applications

#17

Post by free-ekanayaka »

bluejekyll wrote: post #11
hardwaresofton wrote: post #8

Earlier quoted context omitted.

Yeah, but it's probably not smart to re-invent the wheel too many times. Especially when the projects that came before don't look badly written or unreasonable. I think I do have something to offer -- mostly trying gossip and making use of aggressive automatic sharding -- but I'm also more interested in what I wanted to build on top of distributed SQLite as well. No need to get down in the weeds if someone's already…

All great points. I was just responding to what seemed like a defeated perspective. By all means contribute to this project. Everything starts somewhere!

I implemented dqlite in Go for convenience (a production-ready raft library was available, github.com/hashicorp/raft). It'd be interesting to implement something similar in C or event batter in rust, so you can link the library from languages than just Go. But not sure there'd be good use cases for that.

Re: Distributed SQLite for Go applications

#18

Post by free-ekanayaka »

notamy wrote: post #2

After looking through the repo, I'm still not sure exactly why you would want / need this. What exactly is the use-case here?

If your application is:

1) Distributed (e.g. n equal processes running on n machines) 2) Needs some shared state across nodes 3) Would like that state to have SQL semantics (relations, transactions, etc.) 4) Not to heavy on writes to this shared state (I might provide some ballpark numbers at some point) 5) Wants to avoid the operational overhead of an external storage system (mysql, postgresql) 6) Wants to be fault-tolerant and have transparent failovers

then dqlite might be a good choice.

Re: Distributed SQLite for Go applications

#19

Post by free-ekanayaka »

throwaway5752 wrote: post #4

How does it compare to RocksDB? edit: I'm sorry, I was thinking of BedrockDB.

BedrockDB requires to operate a separate process, whereas you can embed dqlite in your Go application (pretty much in the SQLite philosophy). Also, afaik BedrockDB patches upstream SQLite with some more intrusive changes than dqlite (e.g. for supporting concurrent writes). The SQLite patch that dqlite requires is pretty minimal and just adds hooks to internal WAL events.

Re: Distributed SQLite for Go applications

#20

Post by hamandcheese »

notamy wrote: post #2

After looking through the repo, I'm still not sure exactly why you would want / need this. What exactly is the use-case here?

I would imagine it would be used similarly to etcd, zookeeper, consul, etc. as a distributed configuration store, with the advantage of being SQL and the schema goodness that comes along with that.
Post reply on HN