Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nobody ever got fired for using a struct

feldera.com

Post reply on HN
Print view View original

111–120 of 140 posts

Re: Nobody ever got fired for using a struct

#111

Post by nchmy »

hdjrudni wrote: post #84
amluto wrote: post #36

There are many systems that take a native data structure in your favorite language and, using some sort of reflection, makes an on-disk structure that resembles it. Python pickles and Java’s serialization system are infamous examples, and rkyv is a less alarming one. I am quite strongly of the opinion that one should essentially never use these for anything that needs to work well at any scale. If you need an industr…

Don't forget PHP's serialize/unserialize, it's also sketchy. Looks like they at least put up a big warning in their docs: https://www.php.net/manual/en/function.unserialize.php Not hating on PHP, to be clear. It has its warts, but it has served me well.

igbinary is often a good drop-in replacement for native serialize/unserialize. Faster and smaller.

Re: Nobody ever got fired for using a struct

#112

Post by john01dav »

amluto wrote: post #36

There are many systems that take a native data structure in your favorite language and, using some sort of reflection, makes an on-disk structure that resembles it. Python pickles and Java’s serialization system are infamous examples, and rkyv is a less alarming one. I am quite strongly of the opinion that one should essentially never use these for anything that needs to work well at any scale. If you need an industr…

> even a really cool library that makes Rust do this.

The first library that comes to mind when I think of this is `serde` with `#[derive(Serialize, Deserialize)]`, but that gives persistence-format output as you describe is preferable to the former case. I usually use it with JSON.

So, this seems like it may be a false dichotomy.

Re: Nobody ever got fired for using a struct

#113

Post by kwillets »

lsuresh wrote: post #98

Feldera co-founder here. Great discussions here. Some folks pointed out that no one should design a SQL schema like this and I agree. We deal with large enterprise customers, and don't control the schemas that come our way. Trust me, we often ask customers if they have any leeway with changing their SQL and their hands are often tied. We're a query engine, so have to be able to ingest data from existing data sources…

This site is underweighted on OLAP. Columnstores were invented for precisely this use case; nobody in the field wants to normalize everything.

Which brings me to the question, why a rowstore? Are Z-sets hard to manage otherwise?

Another aspect of wide tables is that they tend to have a lot of dependencies, ie different columns come from different aggregations, and the whole table gets held up if one of them is late. IVM seems like a good solution for that problem.

Re: Nobody ever got fired for using a struct

#114

Post by zadikian »

vlovich123 wrote: post #103
imtringued wrote: post #96

Earlier quoted context omitted.

Actually, it's you who is giving that impression with an ultra vague "doesn't solve the problems described". The only problem in the blog post is efficient coding of optional fields and all they was introduce a bitmap. From that perspective, JSON and XML solve the optional fields problem to perfection, since an absent field costs exactly nothing.

I guess you missed the part where the size of the data stored on disk and efficient deserialization are also critically important performance characteristics that neither JSON nor XML have? Capnproto doesn’t support transform on serialize - the optional fields still take up disk space unless you use the packed representation which has some performance drawbacks. Also the generated capnproto rust code is quite heavy o…

Indeed Capnproto is more optimized for serdes time than space usage.

Re: Nobody ever got fired for using a struct

#115

Post by zadikian »

amluto wrote: post #107
vlovich123 wrote: post #104

Earlier quoted context omitted.

The space overhead and the overhead of serialization/deserialization. Rkyv is zero overhead - it’s random access without needing to deserialize and can even be memory mapped.

The whole “zero overhead” thing is IMO a red herring. I care about a few things: stability across versions and languages, space efficiency (sometimes) and performance. I do not care about “overhead” — performance trumps overhead every time. Your deserializer is probably running on a CPU, and that CPU probably has a very fast L1 cache and might be targeted by a compiler that can do scalar replacement of aggregates and…

Serdes time can be significant. There are use cases for the zero copy formats even though they use more space. Likewise bit-packed asn1 is often slower than byte-aligned.

Re: Nobody ever got fired for using a struct

#116

Post by zadikian »

pjc50 wrote: post #91
userbinator wrote: post #76

Earlier quoted context omitted.

and often performance as well BS. Nothing can be faster than a read()/write() (or even mmap()) into a struct, because everything else would need to do more work.

Sure, if your structure doesn't contain any pointers and you only ever want to support one endianness and you trust your compiler to fix the machine layout of the struct forever.

Mainly the first thing. If your struct is already serial, of course serialization will be easy.

Re: Nobody ever got fired for using a struct

#117

Post by userbinator »

pjc50 wrote: post #91
userbinator wrote: post #76

Earlier quoted context omitted.

and often performance as well BS. Nothing can be faster than a read()/write() (or even mmap()) into a struct, because everything else would need to do more work.

Sure, if your structure doesn't contain any pointers and you only ever want to support one endianness and you trust your compiler to fix the machine layout of the struct forever.

...which is true for 99.999% of the time anyway, so it's not worth worrying about.

Re: Nobody ever got fired for using a struct

#118

Post by cozzyd »

vlovich123 wrote: post #104
locknitpicker wrote: post #75

Earlier quoted context omitted.

> Protobufs definitely doesn’t solve the problems described. Capnproto may solve it but I’m not 100% sure. JSON/XML/ASN.1 definitely don’t. I'm not sure you are serious. What open problem do you have in mind? Support for persisting and deserializing optional fields? Mapping across data types? I mean, some JSON deserializers support deserializing sparse objects even to dictionaries. In .NET you can even deserialize ra…

The space overhead and the overhead of serialization/deserialization. Rkyv is zero overhead - it’s random access without needing to deserialize and can even be memory mapped.

If you care about space, you're almost certainly going to compress your output (unless, like, you're literally storing random noise) and so you'll necessarily have overhead from that.

Unless the reason you care about space is because it's some sort of wire protocol for a slow network (like LoRaWAN or Iridium packets or a binary UART protocol), where compression probably doesn't make sense because the compression overhead is too large. But even here, just defining the data layout makes sense, I think.

Tihs could take the form of a C struct with __attribute__((packed)) but that is fragile if you care about more platforms than one. (I generally don't, so that works for me!).

Re: Nobody ever got fired for using a struct

#119

Post by kristianp »

> A new use case processed about the same amount of data as their existing pipelines, but it ran much slower.

Did I miss something? They didn't mention why the new use case was slower. I was expecting some callback to that new usecase in the article somewhere.

Always enjoy reading about performance debugging, thanks for writing this.

Edit: they didn't talk about profiling either. It was an enjoyable read of rust serialisation for non rusty people though.

Re: Nobody ever got fired for using a struct

#120

Post by amluto »

john01dav wrote: post #112
amluto wrote: post #36

There are many systems that take a native data structure in your favorite language and, using some sort of reflection, makes an on-disk structure that resembles it. Python pickles and Java’s serialization system are infamous examples, and rkyv is a less alarming one. I am quite strongly of the opinion that one should essentially never use these for anything that needs to work well at any scale. If you need an industr…

> even a really cool library that makes Rust do this. The first library that comes to mind when I think of this is `serde` with `#[derive(Serialize, Deserialize)]`, but that gives persistence-format output as you describe is preferable to the former case. I usually use it with JSON. So, this seems like it may be a false dichotomy.

Maybe a little bit. But serde works with JSON (among other formats), and you can use it to read and write JSON that interoperates with other libraries and languages just fine. Kind of like how SQLAlchemy looks kind of like you’re writing normal Python code, but it interoperates with SQL.

rkyv is not like this.

Post reply on HN