I sped up serde_json strings by 20% • Hacker News Board

Re: I sped up serde_json strings by 20%

#101

Post by evilduck » Sat, Aug 24, 2024, 5:08 PM UTC

runevault wrote: post #32

echelon wrote: post #29

Earlier quoted context omitted.

> The dependency trees for any meaty Rust project quickly become pretty horrifying. s/Rust// This is really no different from any other language. At least Rust, with Cargo, makes it easy to scan your dependencies. And many notable Rust projects attempt to keep third party dependencies to a minimum. C++ gives you absolutely nothing to work with. Other languages with package managers don't keep dependency trees shallow…

I'm not sure languages without a package manager being the default are nearly as bad. When it is trivial (like running a single command line or a single line in a file like cargo.toml), having a proliferation of dependencies is so easy it almost becomes guaranteed. But in languages like c++ where not everyone uses them (even if several PMs exist) adding a dependency to a library is a much bigger deal as you either ha…

> I'm not sure languages without a package manager being the default are nearly as bad.

Javascript doesn't have a specified package manager.

Re: I sped up serde_json strings by 20%

#102

Post by kelnos » Sat, Aug 24, 2024, 5:34 PM UTC

IshKebab wrote: post #52

purplesyringa wrote: post #12

Earlier quoted context omitted.

> Rust should have Jason built in. I don't think this is a reasonable approach. That's just a way to introduce bloat. Importantly, std does not differ from other crates, except for stability guarantees, so there would be no positive here. All it does is link the library's release cycle to the compiler's. (In fact, rustc-serialize used to be built-in, so Rust tried to go that way.) But also, serde_json isn't large by…

Std definitely differs from other crates: 1. There's only one version so you can't end up with multiple copies of the crate. 2. It is precompiled, so it doesn't bloat your target directory or compile time. 3. It is able to use unstable features without using the nightly compiler. It's a totally reasonable approach. Many other languages have JSON support in their standard libraries and it works fine. I'm not sure I'd…

> Many other languages have JSON support in their standard libraries and it works fine. I'm not sure I'd want it, but I wouldn't say it's an obviously bad idea.

I would say it's a bad idea. JSON is, for lack of a better (less derogatory) term, a data-format fad. If Rust had been designed back in 2000 we'd be having this discussion about XML. Hell, consider Javascript (where JSON comes from), with XHR: remember that stands for "XMLHttpRequest"! Of course it can be used with data payloads other than XML; fortunately the people who added it weren't that short-sighted, but the naming is an interesting historical artifact that shows what was only fleetingly dominant at the time as an API data format.

In another 20 years, when Rust is hopefully still a relevant, widely-used language, we may not be using JSON much at all (and oof, I really hope we aren't), and yet the Rust team would still have to maintain that code were it in the stdlib.

Consider also that Rust's stdlib doesn't even have a TOML parser, even though that seems to be Rust's configuration format of choice.

Re: I sped up serde_json strings by 20%

#103

Post by kelnos » Sat, Aug 24, 2024, 5:38 PM UTC

wiseowise wrote: post #49

purplesyringa wrote: post #12

Earlier quoted context omitted.

> Rust should have Jason built in. I don't think this is a reasonable approach. That's just a way to introduce bloat. Importantly, std does not differ from other crates, except for stability guarantees, so there would be no positive here. All it does is link the library's release cycle to the compiler's. (In fact, rustc-serialize used to be built-in, so Rust tried to go that way.) But also, serde_json isn't large by…

> I don't think this is a reasonable approach. That's just a way to introduce bloat. Can this meme die already? The fact that out of the box install of Rust can’t parse JSON is a joke, and you know it.

If Rust were a "web language", sure, I'd think it would have to have JSON support built in.

Rust is a systems programming language. If Rust had JSON support built in, I'd take it much less seriously. JSON is a fad, just like XML was 20 years ago. In 20 years, when JSON goes the way of XML, the Rust stdlib team should not have to continue maintaining a JSON parser.

An out of the box install of C can't parse JSON either. Do you think C is a joke? C++? Java?

Re: I sped up serde_json strings by 20%

#104

Post by ModernMech » Sat, Aug 24, 2024, 5:40 PM UTC

wiseowise wrote: post #98

ModernMech wrote: post #92

Earlier quoted context omitted.

Based on your other reply about JSON being a "basic" feature I assume you do a lot of work with JSON. What you need to understand is not everyone works with JSON, and for them it's a feature to not have JSON parsing code in their binaries. It's not a loss for them.

Where did you get a notion that JSON parsing code will end up in a binary if it’s not used? Or Rust compiler is so obtuse it can’t tree shake unused code?

How did you get that from what I said? JSON isn't included in Rust binaries because of its ability to bring in only what's needed, and my ability as a developer to specify that as a fine-gained level at compilation time.

Using a language where you don't bring things in as needed means they're built-in at the interpreter level or some other scheme like a large required standard library.

Maybe in those languages your compiler is smart enough to filter out unused code, maybe you don't even have a compiler and you have to distribute your code with an interpreter or a virtual machine with all the batteries. Either way, languages where libs are brought in as-needed are at an advantage when it comes to compiler tech to generate such binaries.

Re: I sped up serde_json strings by 20%

#105

Post by kelnos » Sat, Aug 24, 2024, 5:41 PM UTC

purplesyringa wrote: post #12

zadokshi wrote: post #3

Serde json has 3gb of dependencies once you do a build for debug and a build for release. Use serde on a few active projects and you run out of disk space. I don’t know why json parsing needs 3gb of dependencies. I’m all for code reuse but Serde for json is a bit of a dogs breakfast when it comes to dependencies. all you need is an exploit in on of those dependencies and half of the rust ecosystem is vulnerable. Rust…

> Rust should have Jason built in. I don't think this is a reasonable approach. That's just a way to introduce bloat. Importantly, std does not differ from other crates, except for stability guarantees, so there would be no positive here. All it does is link the library's release cycle to the compiler's. (In fact, rustc-serialize used to be built-in, so Rust tried to go that way.) But also, serde_json isn't large by…

> That's just a way to introduce bloat.

I don't think "bloat" is the issue; as I'm sure you know, Rust programs only contain code for the features of the stdlib they use; if it had JSON support and it wasn't used, the linker would omit that code from the final binary. (Granted, that would make the linker work a little harder.)

More at issue is maintenance burden. Data formats come and go over time. In 10 or 20 years when JSON has faded to XML's level of relevance, the Rust stdlib team shouldn't have to continue maintaining a JSON parser.

Re: I sped up serde_json strings by 20%

#106

Post by kelnos » Sat, Aug 24, 2024, 5:52 PM UTC

wiseowise wrote: post #48

echelon wrote: post #29

Earlier quoted context omitted.

> The dependency trees for any meaty Rust project quickly become pretty horrifying. s/Rust// This is really no different from any other language. At least Rust, with Cargo, makes it easy to scan your dependencies. And many notable Rust projects attempt to keep third party dependencies to a minimum. C++ gives you absolutely nothing to work with. Other languages with package managers don't keep dependency trees shallow…

And yet with Python and JS I don’t need to pull a freaking third-party library for something as basic as JSON.

Most of my Rust projects don't need a JSON parser. You probably haven't been a programmer that long if you think JSON is "basic". If it was 20 years ago, you'd be arguing for an XML parser in Rust's stdlib. In 20 years I'm sure it will be something else. File/data formats come and go. The stdlib of a systems programming language shouldn't be taking on a forever maintenance burden for something that likely won't be in widespread use for all that long.

Re: I sped up serde_json strings by 20%

#107

Post by wtetzner » Sat, Aug 24, 2024, 7:51 PM UTC

inferiorhuman wrote: post #24

tredre3 wrote: post #23

Earlier quoted context omitted.

I arrive at almost the same result as you, with 76MB. I've also checked .cargo, .rustup, and my various cache folders (just in case) and haven't found any additional disk usage. OP is clearly mistaken.

The first thing that jumps out is that the code example doesn't work. The next thing is that the example merely calls cargo build. Using an IDE of any sort will typically invoke rust-analyzer which will bloat the target directory quite a bit. I've also found that stale build artifacts tend to chew up a lot of space (especially if you're trying to measure the typically smaller release builds). Beyond that, none of the…

> So yeah a minimal example won't use a lot of space but if you start to use the bells and whistles serde brings you will definitely bloat your target directory.

Which seems orthogonal to the number of dependencies?

Re: I sped up serde_json strings by 20%

#108

Post by maccard » Sat, Aug 24, 2024, 8:02 PM UTC

pezezin wrote: post #82

maccard wrote: post #61

Earlier quoted context omitted.

I’m on mobile so I can’t check at the moment. But I’d be shocked if the equivalent go binary was anywhere near as big, or took anywhere near as long to build. I’ll check later

I don't know what you consider big and long, but on my computer (Ryzen 5700X) it took 7 seconds to build, and the resulting binary is 556 kB.

I'm on an M1 mac, and I followed the instructions above (cargo build --release && du -sh target) and it took 6 seconds, and the target dir is 35MB. I ran it twice to make sure I wasn't pulling the remote dependencies

go.mod

    module json-test

    go 1.21.1

main.go package main

    import (
        "encoding/json"
        "fmt"
    )

    type Foo struct {
        Bar string `json:"bar"`
    }

    func main() {
        var foo Foo

        json.Unmarshal([]byte(`{"bar": "Hello, World!"}`), &foo)

        fmt.Println(foo.Bar)
    }

time go build && du -sh .

    go build  0.07s user 0.11s system 348% cpu 0.051 total
    2.4M .

I'd say 15x larger and 12x slower "bigger and longer" at least.

Re: I sped up serde_json strings by 20%

#109

Post by tialaramex » Sat, Aug 24, 2024, 9:14 PM UTC

wtetzner wrote: post #75

wiseowise wrote: post #49

Earlier quoted context omitted.

> I don't think this is a reasonable approach. That's just a way to introduce bloat. Can this meme die already? The fact that out of the box install of Rust can’t parse JSON is a joke, and you know it.

Out-of-the-box you can add serde_json to your Cargo.toml file in a single line and have JSON parsing. serde_json = "*" I'm not sure I see the problem.

If you have a vaguely modern Rust, you can just "cargo add serde_json" and Cargo will make the change for you.

Re: I sped up serde_json strings by 20%

#110

Post by IshKebab » Sat, Aug 24, 2024, 9:28 PM UTC

kelnos wrote: post #102

IshKebab wrote: post #52

Earlier quoted context omitted.

Std definitely differs from other crates: 1. There's only one version so you can't end up with multiple copies of the crate. 2. It is precompiled, so it doesn't bloat your target directory or compile time. 3. It is able to use unstable features without using the nightly compiler. It's a totally reasonable approach. Many other languages have JSON support in their standard libraries and it works fine. I'm not sure I'd…

> Many other languages have JSON support in their standard libraries and it works fine. I'm not sure I'd want it, but I wouldn't say it's an obviously bad idea. I would say it's a bad idea. JSON is, for lack of a better (less derogatory) term, a data-format fad. If Rust had been designed back in 2000 we'd be having this discussion about XML. Hell, consider Javascript (where JSON comes from), with XHR: remember that s…

Ha I really hope you're right about JSON being a fad, and something better will come along.

I would bet against it though. JSON's flaws (ambiguous spec re numbers & duplicate keys etc; no comments, strings aren't zero copy...) are pretty minor compared to XML's (completely wrong data model, insanely verbose, not even basic data types).

There was much more motivation to replace XML than JSON.

Also even though XML has been broadly replaced, it's still widespread and I don't think it would be out of place to have it in a standard library. Go has, and Go's standard library is one of its highlights.