Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dqlite – High-Availability SQLite

dqlite.io

Post reply on HN
Print view View original

81–90 of 120 posts

Re: Dqlite – High-Availability SQLite

#81

Post by SomeOldThrow »

stefan_ wrote: post #73
SomeOldThrow wrote: post #64

Earlier quoted context omitted.

I don't want to flame, but I did find it curious they went with C rather than Rust. In my experience the transition is straightforward and the string handling (particularly with unicode encodings) is way better (in addition to the normal ownership benefits), and the result (an easily linkable library exposing a C ABI) is roughly the same.

Yes, and one library will be 100 MiB, the other 1.

A) disk space seems like a reasonable tradeoff for many situations, especially when binaries typically aren't the source of data consumption

B) I don't think I've seen a rust binary more than 10 megs. Cargo, rust, and ripgrep are both about 6 mb on my disk; fd is 2.5mb. These seem like a reasonable standins for a binary of significant size and complexity. sqlite3 itself is about 1.3mb.

C) Dqlite doesn't seem particularly concerned with disk-constrained systems, though I may be interpreting their site incorrectly, and the low footprint should be equally achievable with the rust runtime—surely the database itself would be a much larger concern.

This just seems like an unusually good fit for the benefits of the language—reliable client glue you can import into many runtimes where being able to prove data flow would be an strong defensive coding pattern. That said, I think that C is a good, conservative approach here, I'm certainly not knocking anyone's judgement. Overall the parent poster is absolutely correct: there's a strong correlation between use of rust's type system and size of outputted code.

EDIT: phrasing.

Re: Dqlite – High-Availability SQLite

#82

Post by Dowwie »

FreeHugs wrote: post #62
Dowwie wrote: post #42

Earlier quoted context omitted.

That's a hell of a reason not to use sqlite. Staging data in a temporary table while a table is dropped, recreated, and then data is reinserted is not much of an inconvenience.

It is. It is a bunch of complex commands. Just look at the proposed solutions on Stackoverflow: https://stackoverflow.com/questions/8442147/how-to-delete-or...

So I'll keep my opinion about this out of this response. To those who are interested, the approved answer on that SO thread recommends:

    1. create new table as the one you are trying to change
    2. copy all data,
    3. drop old table,
    4. rename the new one.
It's up to others to decide whether that is complex enough to warrant avoiding sqlite

Re: Dqlite – High-Availability SQLite

#83

Post by lima »

free-ekanayaka wrote: post #53
bryanlarsen wrote: post #20

Has anybody run the Jepsen distributed database tests against dqlite?

Not that I know, but it's on my todo list (dqlite author here).

Does sqlite have consistent reads (i.e. are reads guaranteed to be served by the current leader)?

Re: Dqlite – High-Availability SQLite

#85

Post by tracker1 »

MuffinFlavored wrote: post #61
bb88 wrote: post #59

Interesting nugget about golang from their FAQ: https://github.com/canonical/dqlite/blob/master/doc/faq.md Why C? The first prototype implementation of dqlite was in Go, leveraging the hashicorp/raft implementation of the Raft algorithm. The project was later rewritten entirely in C because of performance problems due to the way Go interoperates with C: Go considers a function call into C that lasts more than ~20 mic…

> since all major languages have provisions to create C bindings. Does WebAssembly or any of its runtimes provide a way to do this?

Yes, emscripten/llvm builds C code into webassembly. I'd say a significant portion of webassembly is C/C++ sourced.

Re: Dqlite – High-Availability SQLite

#86

Post by tracker1 »

SomeOldThrow wrote: post #64
bb88 wrote: post #59

Interesting nugget about golang from their FAQ: https://github.com/canonical/dqlite/blob/master/doc/faq.md Why C? The first prototype implementation of dqlite was in Go, leveraging the hashicorp/raft implementation of the Raft algorithm. The project was later rewritten entirely in C because of performance problems due to the way Go interoperates with C: Go considers a function call into C that lasts more than ~20 mic…

I don't want to flame, but I did find it curious they went with C rather than Rust. In my experience the transition is straightforward and the string handling (particularly with unicode encodings) is way better (in addition to the normal ownership benefits), and the result (an easily linkable library exposing a C ABI) is roughly the same.

I think it's pragmatism. There working two libraries that are already C. Also, a mentioned in another thread, type data couple bloat things a little, I think the specific statement is hyperbolic tough.

My guess is their main goal was to use this in go, where they already have experienced go and C developers and adding a third language would muddy things.

Re: Dqlite – High-Availability SQLite

#87

Post by nickpsecurity »

bb88 wrote: post #59

Interesting nugget about golang from their FAQ: https://github.com/canonical/dqlite/blob/master/doc/faq.md Why C? The first prototype implementation of dqlite was in Go, leveraging the hashicorp/raft implementation of the Raft algorithm. The project was later rewritten entirely in C because of performance problems due to the way Go interoperates with C: Go considers a function call into C that lasts more than ~20 mic…

They could also use D or Rust for this. If borrow-checker is too much, Rust can still do automatic, memory management with other benefits remaining. Both also support letting specific modules be unsafe where performance is critical.

Re: Dqlite – High-Availability SQLite

#88

Post by reacharavindh »

nightfly wrote: post #19
usgroup wrote: post #14

I'm not brimming with ideas of where this would be a good fit tbh. Is anyone using it or considering it for anything?

Depending on some details, it would be perfect for storing state for a libvirt cluster management tool I'm working on. Concerns are I can read the data on disk using the regular sqlite3 cli tool, and lack of rust bindings.

I’m curious about the libvirt management tool ypu’re Working on. Any chance it is open source?

Re: Dqlite – High-Availability SQLite

#89

Post by rdtsc »

SomeOldThrow wrote: post #64
bb88 wrote: post #59

Interesting nugget about golang from their FAQ: https://github.com/canonical/dqlite/blob/master/doc/faq.md Why C? The first prototype implementation of dqlite was in Go, leveraging the hashicorp/raft implementation of the Raft algorithm. The project was later rewritten entirely in C because of performance problems due to the way Go interoperates with C: Go considers a function call into C that lasts more than ~20 mic…

I don't want to flame, but I did find it curious they went with C rather than Rust. In my experience the transition is straightforward and the string handling (particularly with unicode encodings) is way better (in addition to the normal ownership benefits), and the result (an easily linkable library exposing a C ABI) is roughly the same.

Well they are patching SQLite and SQLite is written in C. So they'd have to maintain a C path, Rust code, and another layer of C API interface for client bindings, if I understood their architecture correctly.

Re: Dqlite – High-Availability SQLite

#90

Post by Someone »

Dowwie wrote: post #82
FreeHugs wrote: post #62

Earlier quoted context omitted.

It is. It is a bunch of complex commands. Just look at the proposed solutions on Stackoverflow: https://stackoverflow.com/questions/8442147/how-to-delete-or...

So I'll keep my opinion about this out of this response. To those who are interested, the approved answer on that SO thread recommends: 1. create new table as the one you are trying to change 2. copy all data, 3. drop old table, 4. rename the new one. It's up to others to decide whether that is complex enough to warrant avoiding sqlite

That gets more cumbersome if the table has indexes (you will have to create them on the new table), and even more cumbersome if foreign keys point to it (you will have to drop them before step 3 and recreate them after step 4)

“Copy all data” also can be difficult if the table has data that the database created that must stay the same because you use it elsewhere. That shouldn’t be a problem with SQLite, as it doesn’t allow rowid as foreign key, but if you use it as a foreign key outside the database, or use the hash of a full row to detect changes, it may still bite you.

It also may mean being offline for a significant amount of time, but that also often is (effectively) the case for databases that support deleting columns

Post reply on HN