Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dqlite – High-Availability SQLite

dqlite.io

Post reply on HN
Print view View original

101–110 of 120 posts

Re: Dqlite – High-Availability SQLite

#101

Post by bb88 »

tracker1 wrote: post #86
SomeOldThrow wrote: post #64

Earlier quoted context omitted.

I don't want to flame, but I did find it curious they went with C rather than Rust. In my experience the transition is straightforward and the string handling (particularly with unicode encodings) is way better (in addition to the normal ownership benefits), and the result (an easily linkable library exposing a C ABI) is roughly the same.

I think it's pragmatism. There working two libraries that are already C. Also, a mentioned in another thread, type data couple bloat things a little, I think the specific statement is hyperbolic tough. My guess is their main goal was to use this in go, where they already have experienced go and C developers and adding a third language would muddy things.

Yeah, I think that's correct too.

But it made me think that a killer app for Rust would be this entire concept done in one embedded DB with bindings for most of the popular languages.

Re: Dqlite – High-Availability SQLite

#104

Post by derefr »

szczepano wrote: post #63
MuffinFlavored wrote: post #61

Earlier quoted context omitted.

> since all major languages have provisions to create C bindings. Does WebAssembly or any of its runtimes provide a way to do this?

you can probably compile c to webassembly using clang/llvm

That's not what the parent was asking. They were asking if WASM has an FFI interface for runtimes that want to not execute the WASM code entirely sandboxed from the host, but rather want to allow you to dlload(2) code into the OS process hosting the WASM interpreter and call it through WASM ops.

Presumably, https://github.com/CraneStation/wasmtime would benefit from such an FFI API being specified.

Re: Dqlite – High-Availability SQLite

#105

Post by rntksi »

sansnomme wrote: post #95
hardwaresofton wrote: post #21

Earlier quoted context omitted.

There's one more big distinction, rqlite's replication is command based [0] where as dqlite is/was WAL frame-based -- so basically one ships the command and the other ships WAL frames. This distinction means that non-deterministic commands (ex. `RANDOM()`) will work differently. It looks like dqlite's documentation has changed -- for some reason frames are no longer mentioned anywhere[2]. So maybe this isn't the case…

Can you give an example of how the specific differences may occur?

If you do command-based replication, an insert or update that uses RANDOM() would have to be handled differently, lest you have differing values due to each member of the cluster evaluating and producing different values. (Anything that is an impure function basically will have that problem)

Re: Dqlite – High-Availability SQLite

#106

Post by MuffinFlavored »

derefr wrote: post #104
szczepano wrote: post #63

Earlier quoted context omitted.

you can probably compile c to webassembly using clang/llvm

That's not what the parent was asking. They were asking if WASM has an FFI interface for runtimes that want to not execute the WASM code entirely sandboxed from the host, but rather want to allow you to dlload(2) code into the OS process hosting the WASM interpreter and call it through WASM ops. Presumably, https://github.com/CraneStation/wasmtime would benefit from such an FFI API being specified.

Precisely. Do you think this is a feature the WebAssembly team should standardize or is it a bad fit for the language?

Re: Dqlite – High-Availability SQLite

#107

Post by zellyn »

Hey, free-ekanayaka, a few more questions for your FAQ if you're still paying attention:

Does this store the entire log for all time? When you bring up a new node, does it replay the entire history? If not, how do you bring up a new node without data?

How does backup/restore work?

How do upgrades work? Is the shared WAL low-level enough that it's 100% stable/compatible between sqlite/dqlite versions? If not, what happens if half your cluster is on the old version while you're upgrading, and sees things it doesn't understand yet?

Is it possible to encrypt node/node traffic? Or can you easily send the node-node traffic over a proxy, like Envoy? How about over a unix domain socket or "@named" unix domain socket (which we use for Envoy here at Square)

Looks awesome, by the way!

Re: Dqlite – High-Availability SQLite

#108

Post by SomeOldThrow »

stefan_ wrote: post #96
SomeOldThrow wrote: post #81

Earlier quoted context omitted.

A) disk space seems like a reasonable tradeoff for many situations, especially when binaries typically aren't the source of data consumption B) I don't think I've seen a rust binary more than 10 megs. Cargo, rust, and ripgrep are both about 6 mb on my disk; fd is 2.5mb. These seem like a reasonable standins for a binary of significant size and complexity. sqlite3 itself is about 1.3mb. C) Dqlite doesn't seem particul…

2.5 MiB is about what you expect the kernel size to be for an embedded device :) Everything is an embedded device nowadays, so for reference, if you buy a WiFi AP today and open it up, you're likely to find a 8 or 16 MiB NOR flash inside, maybe a 128 MiB NAND flash (with realistically 64 MiB space since it will be doing A/B updates). I don't think the database size is a big concern. For me the focus in dqlite is very…

For an embedded database, they could certainly emphasize that more.

Re: Dqlite – High-Availability SQLite

#109

Post by tylerhou »

mbreese wrote: post #97
bfrog wrote: post #91

Earlier quoted context omitted.

Which adds up to dozens to hundreds of lines of code which needs to be maintained in each project you use sqlite with, vs the one liner of sql that would be required if using a sql that supported it.

It's only required if you need the ability to change the schema of a SQLite table at runtime. I'd wager that's not a very common use-case for SQLite.

What about database migrations? My app which uses a SQLite database needs to store an additional column for a table. Now I have to write a bunch of custom code to migrate it.

Re: Dqlite – High-Availability SQLite

#110

Post by stjohnswarts »

stefan_ wrote: post #73
SomeOldThrow wrote: post #64

Earlier quoted context omitted.

I don't want to flame, but I did find it curious they went with C rather than Rust. In my experience the transition is straightforward and the string handling (particularly with unicode encodings) is way better (in addition to the normal ownership benefits), and the result (an easily linkable library exposing a C ABI) is roughly the same.

Yes, and one library will be 100 MiB, the other 1.

That's a gross exaggeration
Post reply on HN