Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dqlite – High-Availability SQLite

dqlite.io

Post reply on HN
Print view View original

91–100 of 120 posts

Re: Dqlite – High-Availability SQLite

#91

Post by bfrog »

Dowwie wrote: post #82
FreeHugs wrote: post #62

Earlier quoted context omitted.

It is. It is a bunch of complex commands. Just look at the proposed solutions on Stackoverflow: https://stackoverflow.com/questions/8442147/how-to-delete-or...

So I'll keep my opinion about this out of this response. To those who are interested, the approved answer on that SO thread recommends: 1. create new table as the one you are trying to change 2. copy all data, 3. drop old table, 4. rename the new one. It's up to others to decide whether that is complex enough to warrant avoiding sqlite

Which adds up to dozens to hundreds of lines of code which needs to be maintained in each project you use sqlite with, vs the one liner of sql that would be required if using a sql that supported it.

Re: Dqlite – High-Availability SQLite

#92

Post by Dowwie »

Someone wrote: post #90
Dowwie wrote: post #82

Earlier quoted context omitted.

So I'll keep my opinion about this out of this response. To those who are interested, the approved answer on that SO thread recommends: 1. create new table as the one you are trying to change 2. copy all data, 3. drop old table, 4. rename the new one. It's up to others to decide whether that is complex enough to warrant avoiding sqlite

That gets more cumbersome if the table has indexes (you will have to create them on the new table), and even more cumbersome if foreign keys point to it (you will have to drop them before step 3 and recreate them after step 4) “Copy all data” also can be difficult if the table has data that the database created that must stay the same because you use it elsewhere. That shouldn’t be a problem with SQLite, as it doesn’…

good points

Re: Dqlite – High-Availability SQLite

#93

Post by josegonzalez »

SomeOldThrow wrote: post #81
stefan_ wrote: post #73

Earlier quoted context omitted.

Yes, and one library will be 100 MiB, the other 1.

A) disk space seems like a reasonable tradeoff for many situations, especially when binaries typically aren't the source of data consumption B) I don't think I've seen a rust binary more than 10 megs. Cargo, rust, and ripgrep are both about 6 mb on my disk; fd is 2.5mb. These seem like a reasonable standins for a binary of significant size and complexity. sqlite3 itself is about 1.3mb. C) Dqlite doesn't seem particul…

> Dqlite is a fast, embedded, persistent SQL database with Raft consensus that is perfect for fault-tolerant IoT and Edge devices.

Seems like, at least for embedded devices, you'd want something as small as possible so as to avoid consuming all available disk (not to say any other language will balloon it significantly or not).

Re: Dqlite – High-Availability SQLite

#94

Post by liability »

free-ekanayaka wrote: post #57
rockwotj wrote: post #31

I hope the claim to being fully async I/O is just a buzz term, as it's no longer supported in SQLite. https://www.sqlite.org/asyncvfs.html

It's not a buzz term. It's really fully async disk I/O. Dqlite does not use SQLite's stock vfs implementation for writing to disk, as it's an entirely different model (based on raft).

> Q: How does dqlite behave during conflict situations? Does Raft select a winning WAL write and any others in flight writes are aborted?

> A: There can’t be a conflict situation. Raft’s model is that only the leader can append new log entries, which translated to dqlite means that only the leader can write new WAL frames. So this means that any attempt to perform a write transaction on a non-leader node will fail with a ErrNotLeader error (and in this case clients are supposed to retry against whoever is the new leader).

Correct me if I'm wrong, but isn't that essentially the same limitation WAL mode in normal sqlite has? With WAL you can have as many reads going on as you like, in parallel to a single write. That seems directly comparable to what the dqlite FAQ says, unless I'm missing something.

Re: Dqlite – High-Availability SQLite

#95

Post by sansnomme »

hardwaresofton wrote: post #21
emmelaich wrote: post #6

I was going to ask what is the difference between this and rqlite, which also uses Raft. Found the answer on Reddit: > rqlite is a full RDBMS application, but dqlite is a library you must link with other code. It's like the difference between MySQL and libsqlite3.so. * https://www.reddit.com/r/golang/comments/8a8h8y/dqlite_distr...

There's one more big distinction, rqlite's replication is command based [0] where as dqlite is/was WAL frame-based -- so basically one ships the command and the other ships WAL frames. This distinction means that non-deterministic commands (ex. `RANDOM()`) will work differently. It looks like dqlite's documentation has changed -- for some reason frames are no longer mentioned anywhere[2]. So maybe this isn't the case…

Can you give an example of how the specific differences may occur?

Re: Dqlite – High-Availability SQLite

#96

Post by stefan_ »

SomeOldThrow wrote: post #81
stefan_ wrote: post #73

Earlier quoted context omitted.

Yes, and one library will be 100 MiB, the other 1.

A) disk space seems like a reasonable tradeoff for many situations, especially when binaries typically aren't the source of data consumption B) I don't think I've seen a rust binary more than 10 megs. Cargo, rust, and ripgrep are both about 6 mb on my disk; fd is 2.5mb. These seem like a reasonable standins for a binary of significant size and complexity. sqlite3 itself is about 1.3mb. C) Dqlite doesn't seem particul…

2.5 MiB is about what you expect the kernel size to be for an embedded device :)

Everything is an embedded device nowadays, so for reference, if you buy a WiFi AP today and open it up, you're likely to find a 8 or 16 MiB NOR flash inside, maybe a 128 MiB NAND flash (with realistically 64 MiB space since it will be doing A/B updates).

I don't think the database size is a big concern. For me the focus in dqlite is very much on the 'd' - you store atomic configuration data in there, it's not about throughput.

Re: Dqlite – High-Availability SQLite

#97

Post by mbreese »

bfrog wrote: post #91
Dowwie wrote: post #82

Earlier quoted context omitted.

So I'll keep my opinion about this out of this response. To those who are interested, the approved answer on that SO thread recommends: 1. create new table as the one you are trying to change 2. copy all data, 3. drop old table, 4. rename the new one. It's up to others to decide whether that is complex enough to warrant avoiding sqlite

Which adds up to dozens to hundreds of lines of code which needs to be maintained in each project you use sqlite with, vs the one liner of sql that would be required if using a sql that supported it.

It's only required if you need the ability to change the schema of a SQLite table at runtime. I'd wager that's not a very common use-case for SQLite.

Re: Dqlite – High-Availability SQLite

#98

Post by superturkey650 »

mysterydip wrote: post #41
FreeHugs wrote: post #39

The one annoying thing about SQLite is that there is no easy way to change the table structure. Adding/Removing/Renaming columns is super complicated and afaik there is no good command line tool that does it for you. That is the primary reason why I do not consider it for new projects. It's just to slow to iterate on.

Would making a new db with the new structure and essentially importing the old data be a reasonably speedy activity?

It is if your table is just a datastore with no foreign key links or indexes.

See another reply in this thread: https://news.ycombinator.com/item?id=20841814

Re: Dqlite – High-Availability SQLite

#99

Post by [deleted] »

stefan_ wrote: post #96
SomeOldThrow wrote: post #81

Earlier quoted context omitted.

A) disk space seems like a reasonable tradeoff for many situations, especially when binaries typically aren't the source of data consumption B) I don't think I've seen a rust binary more than 10 megs. Cargo, rust, and ripgrep are both about 6 mb on my disk; fd is 2.5mb. These seem like a reasonable standins for a binary of significant size and complexity. sqlite3 itself is about 1.3mb. C) Dqlite doesn't seem particul…

2.5 MiB is about what you expect the kernel size to be for an embedded device :) Everything is an embedded device nowadays, so for reference, if you buy a WiFi AP today and open it up, you're likely to find a 8 or 16 MiB NOR flash inside, maybe a 128 MiB NAND flash (with realistically 64 MiB space since it will be doing A/B updates). I don't think the database size is a big concern. For me the focus in dqlite is very…

[deleted]

Re: Dqlite – High-Availability SQLite

#100

Post by SomeOldThrow »

josegonzalez wrote: post #93
SomeOldThrow wrote: post #81

Earlier quoted context omitted.

A) disk space seems like a reasonable tradeoff for many situations, especially when binaries typically aren't the source of data consumption B) I don't think I've seen a rust binary more than 10 megs. Cargo, rust, and ripgrep are both about 6 mb on my disk; fd is 2.5mb. These seem like a reasonable standins for a binary of significant size and complexity. sqlite3 itself is about 1.3mb. C) Dqlite doesn't seem particul…

> Dqlite is a fast, embedded, persistent SQL database with Raft consensus that is perfect for fault-tolerant IoT and Edge devices. Seems like, at least for embedded devices, you'd want something as small as possible so as to avoid consuming all available disk (not to say any other language will balloon it significantly or not).

I missed that it was primarily intended for embedded devices.
Post reply on HN