Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nobody ever got fired for using a struct

feldera.com

Post reply on HN
Print view View original

121–130 of 140 posts

Re: Nobody ever got fired for using a struct

#121

Post by bombela »

imtringued wrote: post #95
amluto wrote: post #36

There are many systems that take a native data structure in your favorite language and, using some sort of reflection, makes an on-disk structure that resembles it. Python pickles and Java’s serialization system are infamous examples, and rkyv is a less alarming one. I am quite strongly of the opinion that one should essentially never use these for anything that needs to work well at any scale. If you need an industr…

Fully agreed. rkyv looks like something that is hyper optimizing for a very niche case, but doesn't actually admit that it is doing so. The use case here is transient data akin to swapping in-memory data to disk. "However, while the former have external schemas and heavily restricted data types, rkyv allows all serialized types to be defined in code and can serialize a wide variety of types that the others cannot." A…

> You won't be able to use the data outside of Rust and you're probably tied to a specific Rust version.

This seems false after reading the book, the doc, and a cursory reading of the source code.

It is definitely independent of rust version. The code make use of repr(C) on struct (field order follows the source code) and every field gets its own alignment (making it independent from the C ABI alignment). The format is indeed portable. It is also versioned.

The schema of the user structs is in Rust code. You can make this work across languages, but that's a lot of work and code to support. And this project appears to be in Rust for Rust.

On a side note, I find the code really easy to understand and follow. In my not so humble opinion, it is carefully crafted for performance while being elegant.

Re: Nobody ever got fired for using a struct

#122

Post by thechao »

john01dav wrote: post #112
amluto wrote: post #36

There are many systems that take a native data structure in your favorite language and, using some sort of reflection, makes an on-disk structure that resembles it. Python pickles and Java’s serialization system are infamous examples, and rkyv is a less alarming one. I am quite strongly of the opinion that one should essentially never use these for anything that needs to work well at any scale. If you need an industr…

> even a really cool library that makes Rust do this. The first library that comes to mind when I think of this is `serde` with `#[derive(Serialize, Deserialize)]`, but that gives persistence-format output as you describe is preferable to the former case. I usually use it with JSON. So, this seems like it may be a false dichotomy.

I know "serde" is a take on "codec" but *rewrite* was right there! Also, as long as I'm whinging about naming? 'print' and 'parse' are five letter p words in a bidirectional relationship. Oh! Oh! push, peek, poke, ... pull! It even makes more sense than pop! And it's four letters!

Re: Nobody ever got fired for using a struct

#123

Post by bombela »

amluto wrote: post #36

There are many systems that take a native data structure in your favorite language and, using some sort of reflection, makes an on-disk structure that resembles it. Python pickles and Java’s serialization system are infamous examples, and rkyv is a less alarming one. I am quite strongly of the opinion that one should essentially never use these for anything that needs to work well at any scale. If you need an industr…

> (OMG I just discovered rkyv_dyn. boggle. Did someone really attempt to reproduce the security catastrophe that is Java deserialization in Rust?

Trusting possibly malicious inputs is an universal problem.

Here is a simple example:

    echo "rm -rf /" > cmd
    sh cmd
And this problem is no different in rkyv than rkvy_dyn or any other serialization format on the planet. The issue is trusting inputs. This is also called a man in the middle attack.

The solution is to add a cryptographic signature to detect tempering.

Re: Nobody ever got fired for using a struct

#124

Post by amluto »

bombela wrote: post #123
amluto wrote: post #36

There are many systems that take a native data structure in your favorite language and, using some sort of reflection, makes an on-disk structure that resembles it. Python pickles and Java’s serialization system are infamous examples, and rkyv is a less alarming one. I am quite strongly of the opinion that one should essentially never use these for anything that needs to work well at any scale. If you need an industr…

> (OMG I just discovered rkyv_dyn. boggle. Did someone really attempt to reproduce the security catastrophe that is Java deserialization in Rust? Trusting possibly malicious inputs is an universal problem. Here is a simple example: echo "rm -rf /" > cmd sh cmd And this problem is no different in rkyv than rkvy_dyn or any other serialization format on the planet. The issue is trusting inputs. This is also called a man…

This is an unhelpful interpretation. With a decent memory-safe parser, it’s perfectly safe [1] to deserialize JSON or (most of) XML [0] protobuf or Cap’n Proto or HTTP requests, etc. Or to query a database containing untrusted data. You need to be careful that you don’t introduce a vulnerability by doing something unwise with the deserialized result, but a good deserializer will safely produce a correctly typed output given any input, and the biggest risk is that the output is excessively large.

But tools like Pickle or Java deserialization or, most likely, rkyv_dyn will happily give you outputs that contain callables and that contain behavior, and the result is not safe to access. (In Python, it’s wildly unsafe to access, as merely reading a field of a Python object calls functions encoded by the class, and the class may be quite dynamic.)

[0] The world is full of infamously dangerous XML parsers. Don’t use them, especially if they’re written in C or C++ or they don’t promise that they will not access the network.

> The solution is to add a cryptographic signature to detect tempering.

If you don’t have a deserializer that works on untrusted input, how do you verify signatures. Also, do you really thing it’s okay to do “sh $cmd” just because you happen to have verified a signature.

> This is also called a man in the middle attack.

I suggest looking up what a man in the middle attack is.

Re: Nobody ever got fired for using a struct

#125

Post by fiddlerwoaroof »

locknitpicker wrote: post #109
fiddlerwoaroof wrote: post #106

Earlier quoted context omitted.

I’m not justifying the design but splitting a table with several billion rows is not a trivial task, especially when ORMs and such are involved. Additionally, it’s easier to get work scheduled to ship a feature than it is to convince the relevant players to complete the swing.

> I’m not justifying the design but splitting a table with several billion rows is not a trivial task, especially when ORMs and such are involved. I don't agree. Let me walk you through the process. - create the new table - follow a basic parallel writes strategy -- update your database consumers to write to the new table without reading from it -- run a batch job to populate the new table with data from the old tabl…

I've done this sort of thing or worked with people doing it. The concept is simple, actually executing can take months.

Re: Nobody ever got fired for using a struct

#126

Post by bombela »

amluto wrote: post #124
bombela wrote: post #123

Earlier quoted context omitted.

> (OMG I just discovered rkyv_dyn. boggle. Did someone really attempt to reproduce the security catastrophe that is Java deserialization in Rust? Trusting possibly malicious inputs is an universal problem. Here is a simple example: echo "rm -rf /" > cmd sh cmd And this problem is no different in rkyv than rkvy_dyn or any other serialization format on the planet. The issue is trusting inputs. This is also called a man…

This is an unhelpful interpretation. With a decent memory-safe parser, it’s perfectly safe [1] to deserialize JSON or (most of) XML [0] protobuf or Cap’n Proto or HTTP requests, etc. Or to query a database containing untrusted data. You need to be careful that you don’t introduce a vulnerability by doing something unwise with the deserialized result, but a good deserializer will safely produce a correctly typed outpu…

Ah, I see the confusion. rkyv_dyn doesn't serialize code. Rust is compiled to machine code. It would be quite a feat to accomplish.

I was a bit confused when you compared it to Python pickle and assumed you were talking about general input validation somehow.

I agree that pickle and similar are profoundly surprising and error prone. I struggle to find any reasonable reason one would want that.

As for the man in middle attack, I meant that if somebody intercepts the serialized form, they can mutate it. And without a cryptographic signature, you wouldn't know.

Re: Nobody ever got fired for using a struct

#127

Post by bombela »

logdahl wrote: post #61

Strictly speaking, Isn't there still a way to express at least one Illegal string in ArchivedString? I'm not sure how to hint to the Rust compiler which values are illegal, but if the inline length (at most 15 characers) is aliased to the pointer string length (assume little-endian), wouldnt {ptr: null, len: 16} and {inline_data: {0...}, len: 16} both technically be an illegal value? I'm not saying this is better tha…

In the code you will find union { {len, relptr}, [u8; 16] }

The length is first. The pointer second. The inline string is terminated with 0xFF. The length is 62 bits out of 64 bits such that a specific pattern is placed in the first byte that utf8 doesn't collide with.

Re: Nobody ever got fired for using a struct

#128

Post by amluto »

bombela wrote: post #126
amluto wrote: post #124

Earlier quoted context omitted.

This is an unhelpful interpretation. With a decent memory-safe parser, it’s perfectly safe [1] to deserialize JSON or (most of) XML [0] protobuf or Cap’n Proto or HTTP requests, etc. Or to query a database containing untrusted data. You need to be careful that you don’t introduce a vulnerability by doing something unwise with the deserialized result, but a good deserializer will safely produce a correctly typed outpu…

Ah, I see the confusion. rkyv_dyn doesn't serialize code. Rust is compiled to machine code. It would be quite a feat to accomplish. I was a bit confused when you compared it to Python pickle and assumed you were talking about general input validation somehow. I agree that pickle and similar are profoundly surprising and error prone. I struggle to find any reasonable reason one would want that. As for the man in middl…

> rkyv_dyn doesn't serialize code. Rust is compiled to machine code.

Java is compiled to bytecode, and Obj-C is compiled to machine code. Yet both Android and iOS have had repeated severe vulnerabilities related to deserializing an object that contains a subobject of an unexpected type that pulls code along with it. It seems to be that rkyv_dyn has exactly the same underlying issue.

Sure, Rust is “safe”, and if all the unsafe code is sufficiently careful, it ought to be impossible to get the type of corruption that results in direct code execution, memory writes, etc. But systems can be fully compromised by semantic errors, too.

If I’m designing a system that takes untrusted input and produces an object of type Thing, I want Thing to be pure data. Once you start allowing an open set of methods on Thing or its subobjects, you have lost control of your own control flow. So doing:

    thing.a.func()
may call a function that wasn’t even written at the time you wrote that line of code or even a function that is only present in some but not all programs that execute that line of code.

Exploiting this is considerably harder than exploiting pickle, but considerably harder is not the same as impossible.

Re: Nobody ever got fired for using a struct

#129

Post by lsuresh »

kwillets wrote: post #113
lsuresh wrote: post #98

Feldera co-founder here. Great discussions here. Some folks pointed out that no one should design a SQL schema like this and I agree. We deal with large enterprise customers, and don't control the schemas that come our way. Trust me, we often ask customers if they have any leeway with changing their SQL and their hands are often tied. We're a query engine, so have to be able to ingest data from existing data sources…

This site is underweighted on OLAP. Columnstores were invented for precisely this use case; nobody in the field wants to normalize everything. Which brings me to the question, why a rowstore? Are Z-sets hard to manage otherwise? Another aspect of wide tables is that they tend to have a lot of dependencies, ie different columns come from different aggregations, and the whole table gets held up if one of them is late.…

Good questions!

Feldera tries to be row- and column-oriented in the respective parts that matter. E.g. our LSM trees only store the set of columns that are needed, and we need to be able to pick up individual rows from within those columns for the different operators.

I don't think we've converged on the best design yet here though. We're constantly experimenting with different layouts to see what performs best based on customer workloads.

Re: Nobody ever got fired for using a struct

#130

Post by lsuresh »

kristianp wrote: post #119

> A new use case processed about the same amount of data as their existing pipelines, but it ran much slower. Did I miss something? They didn't mention why the new use case was slower. I was expecting some callback to that new usecase in the article somewhere. Always enjoy reading about performance debugging, thanks for writing this. Edit: they didn't talk about profiling either. It was an enjoyable read of rust seri…

We use an in-product profiler (discussed here: https://www.feldera.com/blog/introducing-feldera's-visual-pr...), along with CPU profiles to identify where in the code we're spending time.
Post reply on HN