Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I sped up serde_json strings by 20%

purplesyringa.moe

Post reply on HN
Print view View original

21–30 of 124 posts

Re: I sped up serde_json strings by 20%

#21

Post by tinrab »

zadokshi wrote: post #3

Serde json has 3gb of dependencies once you do a build for debug and a build for release. Use serde on a few active projects and you run out of disk space. I don’t know why json parsing needs 3gb of dependencies. I’m all for code reuse but Serde for json is a bit of a dogs breakfast when it comes to dependencies. all you need is an exploit in on of those dependencies and half of the rust ecosystem is vulnerable. Rust…

From crates.io, `serde` is a 76.4 KiB dependency. And from what I've seen looking through the code, it's pretty minimal.

Re: I sped up serde_json strings by 20%

#22

Post by makeitshine »

INGSOCIALITE wrote: post #17
38 wrote: post #8

Earlier quoted context omitted.

this 100%. serde is a bloated monster, its sad that its the popular JSON, because all it does is make Rust look bad in my opinion. here are some smaller options: https://lib.rs/crates/humphrey_json https://lib.rs/crates/rust_json https://lib.rs/crates/sj

can rust use the json-c library?

I'd assume you could use bindgen and create bindings no problem.

Re: I sped up serde_json strings by 20%

#23

Post by tredre3 »

hermanradtke wrote: post #6
zadokshi wrote: post #3

Serde json has 3gb of dependencies once you do a build for debug and a build for release. Use serde on a few active projects and you run out of disk space. I don’t know why json parsing needs 3gb of dependencies. I’m all for code reuse but Serde for json is a bit of a dogs breakfast when it comes to dependencies. all you need is an exploit in on of those dependencies and half of the rust ecosystem is vulnerable. Rust…

Please show your work. I cannot reproduce "3gb of dependencies". Here is my test: Cargo.toml [package] name = "serde-test" version = "0.1.0" edition = "2021" [dependencies] serde = { version = "1.0.208", features = ["derive"] } serde_json = "1.0.127" src/main.rs use serde::Deserialize; #[derive(Deserialize)] struct Foo { bar: String, } fn main() { let foo: Foo = serde_json::from_str("\"bar\": \"baz\"").unwrap(); prin…

I arrive at almost the same result as you, with 76MB.

I've also checked .cargo, .rustup, and my various cache folders (just in case) and haven't found any additional disk usage.

OP is clearly mistaken.

Re: I sped up serde_json strings by 20%

#24

Post by inferiorhuman »

tredre3 wrote: post #23
hermanradtke wrote: post #6

Earlier quoted context omitted.

Please show your work. I cannot reproduce "3gb of dependencies". Here is my test: Cargo.toml [package] name = "serde-test" version = "0.1.0" edition = "2021" [dependencies] serde = { version = "1.0.208", features = ["derive"] } serde_json = "1.0.127" src/main.rs use serde::Deserialize; #[derive(Deserialize)] struct Foo { bar: String, } fn main() { let foo: Foo = serde_json::from_str("\"bar\": \"baz\"").unwrap(); prin…

I arrive at almost the same result as you, with 76MB. I've also checked .cargo, .rustup, and my various cache folders (just in case) and haven't found any additional disk usage. OP is clearly mistaken.

The first thing that jumps out is that the code example doesn't work.

The next thing is that the example merely calls cargo build. Using an IDE of any sort will typically invoke rust-analyzer which will bloat the target directory quite a bit. I've also found that stale build artifacts tend to chew up a lot of space (especially if you're trying to measure the typically smaller release builds).

Beyond that, none of the serde features that will tend to generate a ton of code are being used.

So yeah a minimal example won't use a lot of space but if you start to use the bells and whistles serde brings you will definitely bloat your target directory. I expect a typical rust project to take around 3–4 gigs for build artifacts depending.

Re: I sped up serde_json strings by 20%

#25

Post by dwattttt »

Sytten wrote: post #20

The utf-8 tricks make me very nervous since I have seen too many attacks with parser confusion. I for with serde for correctness not speed. I hope this was fuzzed all the way with a bunch of invalid utf-8 strings.

Luckily utf-8 structure is _very_ trivial compared to the average parser. Not to say there can't be bugs, but that the internal states of a parser shouldn't be large, and can be exhaustively tested.

Re: I sped up serde_json strings by 20%

#26

Post by hermanradtke »

inferiorhuman wrote: post #24
tredre3 wrote: post #23

Earlier quoted context omitted.

I arrive at almost the same result as you, with 76MB. I've also checked .cargo, .rustup, and my various cache folders (just in case) and haven't found any additional disk usage. OP is clearly mistaken.

The first thing that jumps out is that the code example doesn't work. The next thing is that the example merely calls cargo build. Using an IDE of any sort will typically invoke rust-analyzer which will bloat the target directory quite a bit. I've also found that stale build artifacts tend to chew up a lot of space (especially if you're trying to measure the typically smaller release builds). Beyond that, none of the…

> The first thing that jumps out is that the code example doesn't work.

Good catch. I forgot the braces. It does not change the target directory size in a significant way.

As for your other comments: sure! We can have a real conversation about rust-analyzer and other serde features (though I am not sure which specific features you are referring to) causing the target directory to increase drastically in size. However, a sensationalist comment that claims the _dependencies_ are 3gb appears to be misleading at best.

Re: I sped up serde_json strings by 20%

#27

Post by hinkley »

Sytten wrote: post #20

The utf-8 tricks make me very nervous since I have seen too many attacks with parser confusion. I for with serde for correctness not speed. I hope this was fuzzed all the way with a bunch of invalid utf-8 strings.

This is the sort of space where I’d like to see a fuzzer.

Re: I sped up serde_json strings by 20%

#28

Post by hinkley »

pornel wrote: post #14
zadokshi wrote: post #3

Serde json has 3gb of dependencies once you do a build for debug and a build for release. Use serde on a few active projects and you run out of disk space. I don’t know why json parsing needs 3gb of dependencies. I’m all for code reuse but Serde for json is a bit of a dogs breakfast when it comes to dependencies. all you need is an exploit in on of those dependencies and half of the rust ecosystem is vulnerable. Rust…

Rust emits unreasonable amount of debug information. It's so freakishly large, I expect it's just a bug. Anything you compile will dump gigabytes into the target folder, but that's not representative of the final product (after stripping the debug info, or at least using a toned-down verbosity setting).

Does it need a more compact representation of its debug info?

Re: I sped up serde_json strings by 20%

#29

Post by echelon »

troad wrote: post #16
zadokshi wrote: post #3

Serde json has 3gb of dependencies once you do a build for debug and a build for release. Use serde on a few active projects and you run out of disk space. I don’t know why json parsing needs 3gb of dependencies. I’m all for code reuse but Serde for json is a bit of a dogs breakfast when it comes to dependencies. all you need is an exploit in on of those dependencies and half of the rust ecosystem is vulnerable. Rust…

Dependency bloat is an issue with Rust in general. The dependency trees for any meaty Rust project quickly become pretty horrifying. Auditing all these dependencies is infeasible, and my level of confidence in a lot of them is fairly low. I worked with Rust for a few years, and with the benefit of a few years' experience, I don't think I'll be touching Rust again until the ecosystem matures a great deal (which will o…

> The dependency trees for any meaty Rust project quickly become pretty horrifying.

s/Rust//

This is really no different from any other language.

At least Rust, with Cargo, makes it easy to scan your dependencies. And many notable Rust projects attempt to keep third party dependencies to a minimum.

C++ gives you absolutely nothing to work with. Other languages with package managers don't keep dependency trees shallow. You're holding Rust up to a standard that nothing meets.

Re: I sped up serde_json strings by 20%

#30

Post by duped »

pornel wrote: post #14
zadokshi wrote: post #3

Serde json has 3gb of dependencies once you do a build for debug and a build for release. Use serde on a few active projects and you run out of disk space. I don’t know why json parsing needs 3gb of dependencies. I’m all for code reuse but Serde for json is a bit of a dogs breakfast when it comes to dependencies. all you need is an exploit in on of those dependencies and half of the rust ecosystem is vulnerable. Rust…

Rust emits unreasonable amount of debug information. It's so freakishly large, I expect it's just a bug. Anything you compile will dump gigabytes into the target folder, but that's not representative of the final product (after stripping the debug info, or at least using a toned-down verbosity setting).

Most of your target folder isn't debug info, but stale build artifacts because Cargo doesn't do any garbage collection.
Post reply on HN