Scylla – Real-Time Big Data Database • Hacker News Board

Re: Scylla – Real-Time Big Data Database

#51

Post by nerpderp82 » Wed, Aug 25, 2021, 7:20 PM UTC

jandrewrogers wrote: post #34

nerpderp82 wrote: post #29

Earlier quoted context omitted.

I always love your take even if I don't agree, SpaceCurve was a phenomenal system, one of the most pragmatic, high performance, easy to use MPP database systems I have ever used. We never met btw, was just a user. But I think you are wrong about Rust not having the right machinery for making high performance dbs. Two examples are Noria and Materialize https://github.com/mit-pdos/noria and it its 50k lines, in the imm…

This kind of reinforces my point though: neither Materialize nor Noria are high-performance database kernels, and they don't need to implement the high-performance I/O structures database kernels have that give Rust problems. Rust works great for server software generally, database kernels are a very specific outlier. It is common in recent database kernel architectures to implement an entire virtual memory system in…

Not sure proves your point, but maybe doesn't disprove your point strongly enough. I am not qualified to argue from experience about how Rust is ideally suited in the ways you think it is not. But from everything I have seen, it can do a whole lot of what C++ is also good at. Rust safety is not all or nothing and a codebase could definitely prioritize ergonomics over correctness.

Two things that I saw in the last couple weeks that might start to sway you.

https://github.com/sslab-gatech/Rudra#readme

GhostCell: Separating Permissions from Data in Rust https://www.youtube.com/watch?v=jIbubw86p0M

Even unsafe Rust can be as ergonomic as C++. But that unsafety can be mediated, moderated and controlled.

Re: Scylla – Real-Time Big Data Database

#52

Post by jstrong » Wed, Aug 25, 2021, 8:55 PM UTC

jandrewrogers wrote: post #36

RussianCow wrote: post #30

Earlier quoted context omitted.

> Consequently, to use Rust in a way that produces equivalent performance requires marking most of the address space as "unsafe". And while you could do this, Rust is currently less expressive than modern C++ for this type of code anyway, so it isn't ergonomic either. Based on my (admittedly limited) experience with Rust, this isn't true. Yes, you'd likely have to use "unsafe" a few times in order to implement a data…

In some databases, you neither have transparent virtual memory (like mmap or swap) nor can your runtime objects be guaranteed to exist in physical memory. In these models, references to your runtime objects are not pointers because a series of DMA operations into your address space may relocate them and your reference may also be on disk somewhere. DMA doesn't understand memory layouts or object models and has its ow…

are there any examples of this technique used in open source projects? I'd be interested to look at the code and see what you mean in greater detail.

Re: Scylla – Real-Time Big Data Database

#53

Post by throwaway81523 » Thu, Aug 26, 2021, 2:59 AM UTC

enedil wrote: post #46

throwaway81523 wrote: post #45

Earlier quoted context omitted.

Seastar is sort of a C++-ification of node.js. Now that C++20 has coroutines, I wonder if those could have been used instead of all that chained method stuff.

Seastar already uses coroutines, however coroutines without Seastar reactor (and all the utilities for IO) are useless by themselves. You still need a way to schedule what's being done when.

Hmm ok I haven't looked at Seastar in a while, but it used to depend on Node-like control inversion where you'd pass an explicit lambda to each action, telling the action what to do next. That meant unwinding the handler for a given event into a bunch of nested lambdas. Coroutine would let you write them in a more traditional sequential style, where you'd have a return to the scheduler whenever something could block. Yes you have to write a layer of async io under everything, but that's how any OS works, more or less.