Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An embedded database written in Rust

github.com

Post reply on HN
Print view View original

21–30 of 74 posts

Re: An embedded database written in Rust

#21

Post by coldtea »

dmitrygr wrote: post #7

How well does that handle the storage disappearing halfway through a write? How well does it handle power being cut halfway through an update of some sort? How well does it handle some of the written blocks actually making it to the disc and others not? How about if the ones that made it were not the first or the last it issued to be written? (For predictable answers to these, and many other complex questions, when d…

>How well does that handle the storage disappearing halfway through a write? How well does it handle power being cut halfway through an update of some sort?

Or the drive catching fire and the user manually deleting all existing backups.

Re: An embedded database written in Rust

#22

Post by krenoten »

fowl2 wrote: post #16
fleetfox wrote: post #9

Earlier quoted context omitted.

What would be the use-case for multi-master clustering in embedded?

"real" serverless - mesh databases! ;p

This is honestly a use case I'm experimenting with using a mix of CRDTs and OT. Our systems are becoming more and more location agnostic and I don't feel that our current data infrastructure is adequate to serve the workloads we're going to be facing as compute migrates to the edge.

Re: An embedded database written in Rust

#23

Post by toolslive »

krenoten wrote: post #11
dmitrygr wrote: post #7

How well does that handle the storage disappearing halfway through a write? How well does it handle power being cut halfway through an update of some sort? How well does it handle some of the written blocks actually making it to the disc and others not? How about if the ones that made it were not the first or the last it issued to be written? (For predictable answers to these, and many other complex questions, when d…

ALICE showed that's not always true with sqlite. Sled is being built with an extreme bias toward reliability over features, but as the readme says, it has some time to go before reaching maturity. The tests are quite good at finding new issues and deterministically replaying them, so you can help bake it in by mining bugs using the default test suite and help it get there.

Most database designers assume that a power failure will only affect writes that are pending. Alas, for SSDs and NVMEs that's not always true. A power failure can cause all kinds of corruption. Long story short: even append-only strategies will not save you.

https://www.usenix.org/system/files/conference/fast13/fast13...

Re: An embedded database written in Rust

#27

Post by spacenick88 »

toolslive wrote: post #23
krenoten wrote: post #11

Earlier quoted context omitted.

ALICE showed that's not always true with sqlite. Sled is being built with an extreme bias toward reliability over features, but as the readme says, it has some time to go before reaching maturity. The tests are quite good at finding new issues and deterministically replaying them, so you can help bake it in by mining bugs using the default test suite and help it get there.

Most database designers assume that a power failure will only affect writes that are pending. Alas, for SSDs and NVMEs that's not always true. A power failure can cause all kinds of corruption. Long story short: even append-only strategies will not save you. https://www.usenix.org/system/files/conference/fast13/fast13...

That paper sounds like problems that can not be worked around in software and need hardware fixes?

Re: An embedded database written in Rust

#28

Post by spacenick88 »

Groxx wrote: post #5
tormeh wrote: post #2

No clustering :( I feel like good multi-master asynchronous and synchronous clustering is truly the frontier in DBs.

clustering and embedded seem like almost opposite ends of the spectrum

Then again embedded databases are a great building block for things like etcd. It also might even make sense to have something like this in process because that would remove quite a few failure modes coming from the client connection and simplify deployment

Re: An embedded database written in Rust

#29

Post by spacenick88 »

jacquesm wrote: post #10
tormeh wrote: post #2

No clustering :( I feel like good multi-master asynchronous and synchronous clustering is truly the frontier in DBs.

You can't cluster an embedded database, this comment makes not sense. Compare with SQLite, not with Orcale or Postgres.

Not really, when the embedded database is allowed to run its own threads and network connections embedding something like etcd makes perfect sense. It removes the failure modes of the client connection and simplifies deployment.

Re: An embedded database written in Rust

#30

Post by krenoten »

jsnell wrote: post #25

Other people have had trouble wringing competitive performance out of Bw-Trees despite heroic optimization efforts [0]. Why is this implementation going to beat other index structures with just a bit of tuning? [0] https://news.ycombinator.com/item?id=17041616

It might not. But the critiques of bw trees in terms of performance that I've seen have not had compelling data in terms of things that matter outside of academia or benchmarking shootouts, like write or space amplification. The bw tree is a cheap thing to abandon after I implement a persistent ART and measure it though. The bwtree is only like 1k of rust on top of the modular pagecache, which is the real heart of the system.
Post reply on HN