Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Serde Rust Framework

serde.rs

Post reply on HN
Print view View original

41–50 of 159 posts

Re: The Serde Rust Framework

#41

Post by Jweb_Guru »

UK-Al05 wrote: post #19
pornel wrote: post #12

Earlier quoted context omitted.

The trick with Serde is that it decodes straight into a native Rust struct. You get most of validation for free (it also nicely takes advantage of Rust enums with data), and struct access is maximally fast. A generic JSON decoder would give you a dynamic structure that can contain anything, and then you'd have to pick it apart.

Most deserializers in typed languages allow you to deserialize straight into structs or typed objects... Serde just doesn't use reflection to do it.

APIs like GSON and Jackson have a lot more footguns, because (1) they attempt to (de)serialize types whether or not they were ever intended to be (de)serialized, (2) they are missing information about important constraints due to losing information at runtime (the excuse usually used here is that "they don't do validation" when in reality they are violating expectations of package modularity and encapsulation in a pretty blunt way). For example, they will happily ignore non-null constraints on your fields. In practice, this is a major annoyance and ends up making you very paranoid about (de)serializing any structure you don't have full control over.

Performance of a lot of the reflection-based parsing APIs also leaves something to be desired, which means that projects with more stringent performance requirements often have to resort to stuff like Avro. This still happens with serde, but much more rarely--besides having more compile-time information at its disposal and needing to allocate less, it also provides relatively straightforward hooks to achieve things like zero-copy deserialization for strings where the whole buffer is available at once.

Re: The Serde Rust Framework

#43

Post by johnfn »

unanswered wrote: post #40
the_duke wrote: post #20

Earlier quoted context omitted.

The only downside is compile time bloat. Serde generates heaps and heaps of generic code. This all gets optimized away to be very efficient, but only once it reaches LLVM. Ever tried working on a crate with hundreds or thousands of de/serializable types? Compile times shoot through the roof really quickly, and serde is often the culprit. The maintainer of serde also created `miniserde` [1] to tackle this problem, whi…

Studies have shown that more than 87% of complaints about rust compile time are because people are enabling serde everywhere and for everything.

These concerns are valid. Data serialization/deserialization is an extremely standard thing to need to be able to do. It's not the user's fault if they use the obvious tool for an obvious job and suddenly have bloated compile times.

Re: The Serde Rust Framework

#44

Post by berkut »

pornel wrote: post #12
UK-Al05 wrote: post #4

Earlier quoted context omitted.

I think in most ecosystems it's easier to use a prebuilt serializer/deserializer than your own format.

The trick with Serde is that it decodes straight into a native Rust struct. You get most of validation for free (it also nicely takes advantage of Rust enums with data), and struct access is maximally fast. A generic JSON decoder would give you a dynamic structure that can contain anything, and then you'd have to pick it apart.

However, if you just want a very minimal subset of a JSON response (i.e. a 'status' string field out of say 20 other values in a map), is it not better to just look for one field, as opposed to have to create an entire struct in Rust to represent the full response?

How does Serde work with optional extra fields? I know you can use Option, but that implies you know they exist - what happens if fields get added in the future silently? I guess the schema changes then, which isn't good, but that might happen?

Re: The Serde Rust Framework

#45

Post by kkwteh »

unanswered wrote: post #40
the_duke wrote: post #20

Earlier quoted context omitted.

The only downside is compile time bloat. Serde generates heaps and heaps of generic code. This all gets optimized away to be very efficient, but only once it reaches LLVM. Ever tried working on a crate with hundreds or thousands of de/serializable types? Compile times shoot through the roof really quickly, and serde is often the culprit. The maintainer of serde also created `miniserde` [1] to tackle this problem, whi…

Studies have shown that more than 87% of complaints about rust compile time are because people are enabling serde everywhere and for everything.

That's great. That means that if the Rust compiler is made faster in this one case it'll make a huge difference across the board.

Re: The Serde Rust Framework

#46

Post by rokob »

berkut wrote: post #44
pornel wrote: post #12

Earlier quoted context omitted.

The trick with Serde is that it decodes straight into a native Rust struct. You get most of validation for free (it also nicely takes advantage of Rust enums with data), and struct access is maximally fast. A generic JSON decoder would give you a dynamic structure that can contain anything, and then you'd have to pick it apart.

However, if you just want a very minimal subset of a JSON response (i.e. a 'status' string field out of say 20 other values in a map), is it not better to just look for one field, as opposed to have to create an entire struct in Rust to represent the full response? How does Serde work with optional extra fields? I know you can use Option , but that implies you know they exist - what happens if fields get added in the…

You can add a catch all field that captures those in a map. You can also choose to ignore things you don’t explicitly mention. It is up to you how strict you want the parsing to be.

So you can define a struct with one status field and the right annotations and you will get exactly what you describe without having to write the code and it will still be almost as fast as doing that parsing yourself.

Re: The Serde Rust Framework

#47

Post by kroltan »

berkut wrote: post #44
pornel wrote: post #12

Earlier quoted context omitted.

The trick with Serde is that it decodes straight into a native Rust struct. You get most of validation for free (it also nicely takes advantage of Rust enums with data), and struct access is maximally fast. A generic JSON decoder would give you a dynamic structure that can contain anything, and then you'd have to pick it apart.

However, if you just want a very minimal subset of a JSON response (i.e. a 'status' string field out of say 20 other values in a map), is it not better to just look for one field, as opposed to have to create an entire struct in Rust to represent the full response? How does Serde work with optional extra fields? I know you can use Option , but that implies you know they exist - what happens if fields get added in the…

As far as I understand, unmapped fields are simply ignored, so you can write your code like

    #[derive(Deserialize)]
    struct Response { status: u64 }

Re: The Serde Rust Framework

#48

Post by nemothekid »

berkut wrote: post #44
pornel wrote: post #12

Earlier quoted context omitted.

The trick with Serde is that it decodes straight into a native Rust struct. You get most of validation for free (it also nicely takes advantage of Rust enums with data), and struct access is maximally fast. A generic JSON decoder would give you a dynamic structure that can contain anything, and then you'd have to pick it apart.

However, if you just want a very minimal subset of a JSON response (i.e. a 'status' string field out of say 20 other values in a map), is it not better to just look for one field, as opposed to have to create an entire struct in Rust to represent the full response? How does Serde work with optional extra fields? I know you can use Option , but that implies you know they exist - what happens if fields get added in the…

An "entire" Rust struct containing a single string field is just a struct with a single field. You don't have to specify the fields you don't need and it's unlikely your handrolled parser would be faster than serde (unless you are using memchr or something, then in that case you aren't parsing JSON).

Re: The Serde Rust Framework

#49

Post by fwip »

xet7 wrote: post #14
ttymck wrote: post #11

Earlier quoted context omitted.

This is interesting, and news to me. Your comment suggests something about Rust imposes this problem on all web frameworks, is that true? If so could you share the root cause?

I'm just newbie in Rust. Does some web framework not use hyper? I'm trying to find web framework that does not have problem with Slow-Loris.

This doesn't solve your problem out of the box, but I think most reverse-proxies have configurations that can protect the services behind them from these sorts of attacks.

Re: The Serde Rust Framework

#50

Post by throwaway894345 »

Jweb_Guru wrote: post #35
UK-Al05 wrote: post #4

Earlier quoted context omitted.

I think in most ecosystems it's easier to use a prebuilt serializer/deserializer than your own format.

Nah, not like it is in Rust. For the vast majority of projects, serde makes parsing / serializing something they don't even have to think about , without having to sacrifice type safety to get there. Until you experience that in a large project with a lot of moving parts and developers, it's hard to appreciate just how many developer cycles you were wasting before on stuff that can be totally automated. It's format-a…

> For the vast majority of projects, serde makes parsing / serializing something they don't even have to think about, without having to sacrifice type safety to get there.

This has not been remotely my experience. Deserializing has a lot of implications for ownership and you get into complicated deserializer trait specifications pretty quickly.

In particular, there doesn’t seem to be a way to deserialize a type that has a reference because you don’t seem to be able to tell serde who should own that memory. Maybe I’m wrong, but I’ve walked through this multiple times with experienced Rust users and no one could get it working.

Serde is impressive, I’m sure, but far from “don’t even have to think about it”.

Post reply on HN