Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dqlite – High-Availability SQLite

dqlite.io

Post reply on HN
Print view View original

61–70 of 120 posts

Re: Dqlite – High-Availability SQLite

#61

Post by MuffinFlavored »

bb88 wrote: post #59

Interesting nugget about golang from their FAQ: https://github.com/canonical/dqlite/blob/master/doc/faq.md Why C? The first prototype implementation of dqlite was in Go, leveraging the hashicorp/raft implementation of the Raft algorithm. The project was later rewritten entirely in C because of performance problems due to the way Go interoperates with C: Go considers a function call into C that lasts more than ~20 mic…

> since all major languages have provisions to create C bindings.

Does WebAssembly or any of its runtimes provide a way to do this?

Re: Dqlite – High-Availability SQLite

#62

Post by FreeHugs »

Dowwie wrote: post #42
FreeHugs wrote: post #39

The one annoying thing about SQLite is that there is no easy way to change the table structure. Adding/Removing/Renaming columns is super complicated and afaik there is no good command line tool that does it for you. That is the primary reason why I do not consider it for new projects. It's just to slow to iterate on.

That's a hell of a reason not to use sqlite. Staging data in a temporary table while a table is dropped, recreated, and then data is reinserted is not much of an inconvenience.

It is. It is a bunch of complex commands. Just look at the proposed solutions on Stackoverflow:

https://stackoverflow.com/questions/8442147/how-to-delete-or...

Re: Dqlite – High-Availability SQLite

#63

Post by szczepano »

MuffinFlavored wrote: post #61
bb88 wrote: post #59

Interesting nugget about golang from their FAQ: https://github.com/canonical/dqlite/blob/master/doc/faq.md Why C? The first prototype implementation of dqlite was in Go, leveraging the hashicorp/raft implementation of the Raft algorithm. The project was later rewritten entirely in C because of performance problems due to the way Go interoperates with C: Go considers a function call into C that lasts more than ~20 mic…

> since all major languages have provisions to create C bindings. Does WebAssembly or any of its runtimes provide a way to do this?

you can probably compile c to webassembly using clang/llvm

Re: Dqlite – High-Availability SQLite

#64

Post by SomeOldThrow »

bb88 wrote: post #59

Interesting nugget about golang from their FAQ: https://github.com/canonical/dqlite/blob/master/doc/faq.md Why C? The first prototype implementation of dqlite was in Go, leveraging the hashicorp/raft implementation of the Raft algorithm. The project was later rewritten entirely in C because of performance problems due to the way Go interoperates with C: Go considers a function call into C that lasts more than ~20 mic…

I don't want to flame, but I did find it curious they went with C rather than Rust. In my experience the transition is straightforward and the string handling (particularly with unicode encodings) is way better (in addition to the normal ownership benefits), and the result (an easily linkable library exposing a C ABI) is roughly the same.

Re: Dqlite – High-Availability SQLite

#65

Post by heavenlyblue »

linsomniac wrote: post #51
peterwwillis wrote: post #46

Some thoughts: A consensus protocol is like 1/50th of what you need for a stable, reliable distributed database, and it's developed by a company, so expect it to be abandoned once they stop developing it. I wouldn't use it at work (yet) but could be fun for personal projects.

I wouldn't paint Canonical with that generalization. It's not unheard of that they have dropped projects, but I wouldn't say it's common. But looks like their primary use is LXD, which doesn't seem to be going anywhere...

But the main point made by the parent is entirely correct: the biggest issue isn’t that of implementing a consensus protocol: the biggest issue is the reconfiguration of the cluster, management of the dead/live nodes, addition of extra nodes for replacement, copying of the data before reconfiguration.

All of that needs tooling.

Re: Dqlite – High-Availability SQLite

#66

Post by zellyn »

free-ekanayaka wrote: post #54
maurodelazeri wrote: post #48

how is the performance running 3 nodes? how many inserts/reads can be done in average ?

Depends on how fast is your disk, what file system and kernel you use, and how low is your network latency. Difficult to predict. But it's basically as fast as it can get given 1) hardware constraints 2) raft consensus. If you want light-speed insert/delete, you could probably don't use the disk at all: as long as a majority of your nodes don't die, you won't lose any data. You can also go somewhere in between and sa…

Do all nodes participate as full RAFT nodes? Or can you have read-only nodes?

Re: Dqlite – High-Availability SQLite

#67

Post by peterwwillis »

linsomniac wrote: post #51
peterwwillis wrote: post #46

Some thoughts: A consensus protocol is like 1/50th of what you need for a stable, reliable distributed database, and it's developed by a company, so expect it to be abandoned once they stop developing it. I wouldn't use it at work (yet) but could be fun for personal projects.

I wouldn't paint Canonical with that generalization. It's not unheard of that they have dropped projects, but I wouldn't say it's common. But looks like their primary use is LXD, which doesn't seem to be going anywhere...

It doesn't really matter who wrote it, it's a trope of corporate software development. A small team makes project X to support project Y. They go through the usual dev + production + maintenance cycle, which takes 2-3 years typically, sometimes 5, after which the team is disbanded/reorganized, and no new dev work happens on the old projects. The project is effectively abandoned at that point, unless it happens to have picked up enough users that "a community" forms and picks up development... but that's rare, because corporations don't want to give development of their project over to randos on the internet, especially if they're still using it. The best case is it would fork, or move to some other org's code repo.

I like to use projects which lots of other projects depend on directly. That way if the main project goes unsupported, all the other projects using it will band together to support a fork. I believe open source that is not created for a company will last much longer. (I like that they rewrote it in C, though; it would probably survive well as a fork if enough people/projects use it)

Re: Dqlite – High-Availability SQLite

#68

Post by otoolep »

emmelaich wrote: post #6

I was going to ask what is the difference between this and rqlite, which also uses Raft. Found the answer on Reddit: > rqlite is a full RDBMS application, but dqlite is a library you must link with other code. It's like the difference between MySQL and libsqlite3.so. * https://www.reddit.com/r/golang/comments/8a8h8y/dqlite_distr...

rqlite creator here.

Yes, exactly, Dqlite is a library, rqlite is a full application.

Re: Dqlite – High-Availability SQLite

#69

Post by otoolep »

Scorpiion wrote: post #9

I was reading through the docs and this FAQ is worth checking out: https://github.com/canonical/dqlite/blob/master/doc/faq.md In includes an answer about the difference with rqlite. To me reading the docs it seems like dqlite has been developed by the team who develops LXD at Canonical as LXD is listed as the biggest user of the project and it says on the authors github that he works at Canonical at/with LXD/LXC. Int…

rqlite creator here.

Yes, good luck to the creators of this project, it looks very interesting and I've been watching it for a few years now.

Re: Dqlite – High-Availability SQLite

#70

Post by peterwwillis »

heavenlyblue wrote: post #65
linsomniac wrote: post #51

Earlier quoted context omitted.

I wouldn't paint Canonical with that generalization. It's not unheard of that they have dropped projects, but I wouldn't say it's common. But looks like their primary use is LXD, which doesn't seem to be going anywhere...

But the main point made by the parent is entirely correct: the biggest issue isn’t that of implementing a consensus protocol: the biggest issue is the reconfiguration of the cluster, management of the dead/live nodes, addition of extra nodes for replacement, copying of the data before reconfiguration. All of that needs tooling.

Yep! And you need someone to help fix bizarre bugs in core that only crop up in your own weird environment. With support that's quick & easy, but otherwise you have to form your own dev team to specialize in it, making it costlier.
Post reply on HN