Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Serde Rust Framework

serde.rs

Post reply on HN
Print view View original

51–60 of 159 posts

Re: The Serde Rust Framework

#51

Post by josephg »

AaronO wrote: post #8

Whilst not classical de/serialization I wrote serde_v8 ( https://github.com/denoland/serde_v8 ), an expressive and ~maximally efficient bijection between v8 & rust. It has powered Deno's op-layer since 1.9 ( https://deno.com/blog/v1.9#faster-calls-into-rust-with-serde... ) and has enabled significant improvements in opcall overhead (close to 100x) whilst also simplifying said op-layer.

Thats fantastic.

I maintain the nodejs bindings for foundationdb. Foundationdb refuses to publish a wire protocol, so the bindings are implemented as native code wrapped by n_api. The code is a rat's nest of calls to methods like `napi_get_value_string_utf8` to parse javascript objects into C. (Eg [1]). As well as being difficult to read and write, I'm sure there's weird bugs lurking somewhere in all that boilerplate code. I've made my error checking a bit easier using macros, but that might have only made things worse.

I'd much prefer all that code to just be in rust. serde-v8 looks way easier to use than all the goopy serialization nonsense I'm doing now. (Though I'd want a serde_napi variant instead of going straight to v8).

[1] https://github.com/josephg/node-foundationdb/blob/c1165539e5...

Re: The Serde Rust Framework

#52

Post by slaymaker1907 »

the_duke wrote: post #20
roca wrote: post #2

serde is one of the best things about Rust in practice. It is more convenient to use serde to serialize/deserialize to some standard format like JSON or YAML than it is to write your own half-baked format and serialization/deserialization code --- even for the simplest tasks --- so you just stop doing the latter. This is a fundamental shift, and very good for your software. "Convenience" here actually covers a lot of…

The only downside is compile time bloat. Serde generates heaps and heaps of generic code. This all gets optimized away to be very efficient, but only once it reaches LLVM. Ever tried working on a crate with hundreds or thousands of de/serializable types? Compile times shoot through the roof really quickly, and serde is often the culprit. The maintainer of serde also created `miniserde` [1] to tackle this problem, whi…

Lack of orphan instances really aren't that bad of a problem in practice. If really necessary, the newtype pattern is very easy to do in Rust and has the benefit of never conflicting if the dependency adds a trait implementation. What is way more important is keeping your crate sizes manageable since IIRC Rust doesn't do any incremental/parallel compilation within a single crate.

Compilation speed isn't just one number. While 5min isn't bad at all for compiling a whole project from scratch including dependencies, 5min is a really long time for incremental builds.

Re: The Serde Rust Framework

#53

Post by [deleted] »

Jweb_Guru wrote: post #35
UK-Al05 wrote: post #4

Earlier quoted context omitted.

I think in most ecosystems it's easier to use a prebuilt serializer/deserializer than your own format.

Nah, not like it is in Rust. For the vast majority of projects, serde makes parsing / serializing something they don't even have to think about , without having to sacrifice type safety to get there. Until you experience that in a large project with a lot of moving parts and developers, it's hard to appreciate just how many developer cycles you were wasting before on stuff that can be totally automated. It's format-a…

[deleted]

Re: The Serde Rust Framework

#55

Post by eyelidlessness »

the_duke wrote: post #25
apendleton wrote: post #22

Earlier quoted context omitted.

If you want a field to be able to be not-set, you just need to specify that by making it an `Option ` type. So like, if the field is a number, make it an `Option ` -- that way you can distinguish between not-set (`None`) and set to zero (`Some(0)`). Edit: whoops, misread which language it was you were frustrated with. Nevermind!

You misunderstood, parent is complaining about Go, not about Rust. In Go you would have to use a pointer to represent Option , which makes code really awkward. Go serialization really has quite a few issues apart from default initialization. Configuration by somewhat weird struct tag strings which are only evaluated/validated at runtime, de/serialization is all done via reflection (unless you want to use code generat…

> In Go you would have to use a pointer to represent Option, which makes code really awkward.

This sounded surprising to me, then I remembered Go doesn’t have generics. I imagine this won’t be as much of an issue when it does?

Re: The Serde Rust Framework

#56

Post by roca »

the_duke wrote: post #20
roca wrote: post #2

serde is one of the best things about Rust in practice. It is more convenient to use serde to serialize/deserialize to some standard format like JSON or YAML than it is to write your own half-baked format and serialization/deserialization code --- even for the simplest tasks --- so you just stop doing the latter. This is a fundamental shift, and very good for your software. "Convenience" here actually covers a lot of…

The only downside is compile time bloat. Serde generates heaps and heaps of generic code. This all gets optimized away to be very efficient, but only once it reaches LLVM. Ever tried working on a crate with hundreds or thousands of de/serializable types? Compile times shoot through the roof really quickly, and serde is often the culprit. The maintainer of serde also created `miniserde` [1] to tackle this problem, whi…

That's true. Pernosco has lots of serialization code and compilation is slower than I'd like. I feel that it's still easily worth it.

Re: The Serde Rust Framework

#57

Post by roca »

pornel wrote: post #17

Because Serde is a common API for pretty much all serialisation formats supported in Rust, it's very easy to try multiple formats to choose the best size/speed you need. JSON too big? You can make it CBOR or Msgpack with a couple of lines. Want faster? Call bincode instead.

Yeah, the format independence is amazing. In Pernosco we use bincode for IPC payloads and when we want to log IPCs we just serialize the payloads to JSON with zero developer effort.

Re: The Serde Rust Framework

#58

Post by curun1r »

roca wrote: post #2

serde is one of the best things about Rust in practice. It is more convenient to use serde to serialize/deserialize to some standard format like JSON or YAML than it is to write your own half-baked format and serialization/deserialization code --- even for the simplest tasks --- so you just stop doing the latter. This is a fundamental shift, and very good for your software. "Convenience" here actually covers a lot of…

The other benefit is that almost all of the Rust ecosystem supports serde, often by way of a feature. So using serde can mean being able to embed a type from a crate in your serializable type or being able to pass your own types to crate functions/methods that serialize/deserialize. It all ends up being interoperable.

Re: The Serde Rust Framework

#59

Post by roca »

throwaway894345 wrote: post #50
Jweb_Guru wrote: post #35

Earlier quoted context omitted.

Nah, not like it is in Rust. For the vast majority of projects, serde makes parsing / serializing something they don't even have to think about , without having to sacrifice type safety to get there. Until you experience that in a large project with a lot of moving parts and developers, it's hard to appreciate just how many developer cycles you were wasting before on stuff that can be totally automated. It's format-a…

> For the vast majority of projects, serde makes parsing / serializing something they don't even have to think about, without having to sacrifice type safety to get there. This has not been remotely my experience. Deserializing has a lot of implications for ownership and you get into complicated deserializer trait specifications pretty quickly. In particular, there doesn’t seem to be a way to deserialize a type that…

We almost always deserialize to owned data, partly because our deserialization source is almost always compressed so the data has to be copied. There are a few cases where we want to serialize with a reference and deserialize to owned, but that's no big deal using custom "with" functions.

We have hit one serde footgun, which is that skip-serializing-if corrupts your data with formats like bincode :-(. https://github.com/serde-rs/serde/issues/1732 really wish that could be fixed.

Re: The Serde Rust Framework

#60

Post by codys »

berkut wrote: post #44
pornel wrote: post #12

Earlier quoted context omitted.

The trick with Serde is that it decodes straight into a native Rust struct. You get most of validation for free (it also nicely takes advantage of Rust enums with data), and struct access is maximally fast. A generic JSON decoder would give you a dynamic structure that can contain anything, and then you'd have to pick it apart.

However, if you just want a very minimal subset of a JSON response (i.e. a 'status' string field out of say 20 other values in a map), is it not better to just look for one field, as opposed to have to create an entire struct in Rust to represent the full response? How does Serde work with optional extra fields? I know you can use Option , but that implies you know they exist - what happens if fields get added in the…

If you only want a single field in json, you define a struct with just that single field. Serde does have the ability to return an error about unknown fields, but it is not enabled by default.

This also informs your second question. If new optional fields in json are added, unless you tell serde to complain about them, it won't, and you can add the new Option at your leisure.

Post reply on HN