Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An embedded database written in Rust

github.com

Post reply on HN
Print view View original

31–40 of 74 posts

Re: An embedded database written in Rust

#31

Post by toolslive »

spacenick88 wrote: post #27
toolslive wrote: post #23

Earlier quoted context omitted.

Most database designers assume that a power failure will only affect writes that are pending. Alas, for SSDs and NVMEs that's not always true. A power failure can cause all kinds of corruption. Long story short: even append-only strategies will not save you. https://www.usenix.org/system/files/conference/fast13/fast13...

That paper sounds like problems that can not be worked around in software and need hardware fixes?

You could work around it (erasure coding, fe), if you had insights on the failure mode specifics, but it's vendor specific and vendors are not exactly forthcoming.

So the only thing you can do is add checksumming schemes that allow you to detect you have been affected.

The paper is from 2013, so the situation might have improved meanwhile (I wouldn't put any money on it)

Re: An embedded database written in Rust

#33

Post by krenoten »

samuell wrote: post #32

Nice with more developments in embedded databases! Would be interesting with a comparison with mentat [1]. Do you plan to support any query langauge such as datalog, like mentat? [1] https://github.com/mozilla/mentat

Yeah, I'm curious about using sled as a more ssd friendly storage engine for mentat. I'm just starting to experiment with datalog implementations, but I think by having harmony between the storage engine, query language, and hardware properties we can make a really compelling stateful systems. If this is something that interests you, I'd love to work with more people on this.

Re: An embedded database written in Rust

#34

Post by krenoten »

toolslive wrote: post #23
krenoten wrote: post #11

Earlier quoted context omitted.

ALICE showed that's not always true with sqlite. Sled is being built with an extreme bias toward reliability over features, but as the readme says, it has some time to go before reaching maturity. The tests are quite good at finding new issues and deterministically replaying them, so you can help bake it in by mining bugs using the default test suite and help it get there.

Most database designers assume that a power failure will only affect writes that are pending. Alas, for SSDs and NVMEs that's not always true. A power failure can cause all kinds of corruption. Long story short: even append-only strategies will not save you. https://www.usenix.org/system/files/conference/fast13/fast13...

Indeed. This is why I aggressively checksum everything and pay particular attention to throwing away all data that was written after any detected corruption during recovery. This is easier with the log-only architecture. It's also totally os and filesystem agnostic. I was happily surprised yesterday when it passed tests on fuchsia :]

Re: An embedded database written in Rust

#35

Post by toolslive »

krenoten wrote: post #34
toolslive wrote: post #23

Earlier quoted context omitted.

Most database designers assume that a power failure will only affect writes that are pending. Alas, for SSDs and NVMEs that's not always true. A power failure can cause all kinds of corruption. Long story short: even append-only strategies will not save you. https://www.usenix.org/system/files/conference/fast13/fast13...

Indeed. This is why I aggressively checksum everything and pay particular attention to throwing away all data that was written after any detected corruption during recovery. This is easier with the log-only architecture. It's also totally os and filesystem agnostic. I was happily surprised yesterday when it passed tests on fuchsia :]

So you might end up throwing away everything. (I know, it's not your fault)

Re: An embedded database written in Rust

#36

Post by fooker »

EugeneOZ wrote: post #26
dajonker wrote: post #15

I see that MVCC is a planned feature. Why would you need MVCC for an embedded database? It seems like unnecessary overhead that conflicts with the performance goals.

In Rust abstractions are free.

If that was so, there wouldn't have been a need for the unsafe mode.

Re: An embedded database written in Rust

#38

Post by judofyr »

jstewartmobile wrote: post #37

I'll take Richard Hipp's C code over just about anyone's Rust code any day of the week. The man is a national treasure!

I'll also take any project that has thousands (if not millions?) of man-hours together with an extremely solid test suite. That doesn't mean other projects aren't worth exploring or won't be useful.

Re: An embedded database written in Rust

#40

Post by PudgePacket »

fooker wrote: post #36
EugeneOZ wrote: post #26

Earlier quoted context omitted.

In Rust abstractions are free.

If that was so, there wouldn't have been a need for the unsafe mode.

You're argument doesn't really make sense. Not all abstractions are workarounds for "unsafe" things. For example, unsafe is needed for dealing with the OS, no amount of abstraction will help you there. At some point you must talk to and accept memory that another process (OS) tells you is correct. I'm not so knowledgeable in Rust so I can't say definitely, but I think you should back up your statement with some (any) examples.
Post reply on HN