Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a high performance JSON parser

dave.cheney.net

Post reply on HN
Print view View original

101–110 of 193 posts

Re: Building a high performance JSON parser

#101

Post by mananaysiempre »

coldtea wrote: post #83
suzzer99 wrote: post #81

Can someone explain to me why JSON can't have comments or trailing commas? I really hope the performance gains are worth it, because I've lost 100s of man-hours to those things, and had to resort to stuff like this in package.json: "IMPORTANT: do not run the scripts below this line, they are for CICD only": true,

It can't have comments because it didn't originally had comments, so now it's too late. And it originally didn't have comments, because Douglas Cockford thought they could be abused for parsing instructions. As for not having trailing commas, it's probably a less intentional bad design choice. That said, if you want commas and comments, and control the parsers that will be used for your JSON, then use JSONC (JSON wit…

Amusingly, it originally did have comments. Removing comments was the one change Crockford ever made to the spec[1].

[1] https://web.archive.org/web/20150105080225/https://plus.goog... (thank you Internet Archive for making Google’s social network somewhat accessible and less than useless)

Re: Building a high performance JSON parser

#102

Post by isuckatcoding »

dh2022 wrote: post #73
isuckatcoding wrote: post #59

Earlier quoted context omitted.

Yes but for applications where you need to do ETL style transformations on large datasets, streaming is an immensely useful strategy. Sure you could argue go isn’t the right tool for the job but I don’t see why it can’t be done with the right optimizations like this effort.

If performance is important why would you keep large datasets in JSON format?

Usually because the downstream service or store needs it

Re: Building a high performance JSON parser

#103

Post by ahoka »

craigching wrote: post #85
coldtea wrote: post #82

Earlier quoted context omitted.

What line of work are you in that you've "written far too many JSON parsers already" in your career?!!!

Probably anywhere that requires parsing large JSON documents. Off the shelf JSON parsers are notoriously slow on large JSON documents.

Not necessarily, for example Newtonsoft is fine with multiple hundreds of megabyes if you use it correctly. But of course depends on how large we are talking about.

Re: Building a high performance JSON parser

#104

Post by mananaysiempre »

tubthumper8 wrote: post #97
coldtea wrote: post #83

Earlier quoted context omitted.

It can't have comments because it didn't originally had comments, so now it's too late. And it originally didn't have comments, because Douglas Cockford thought they could be abused for parsing instructions. As for not having trailing commas, it's probably a less intentional bad design choice. That said, if you want commas and comments, and control the parsers that will be used for your JSON, then use JSONC (JSON wit…

Does JSONC have a specification or formal definition? People have suggested[1] using JSON5[2] instead for that reason [1] https://github.com/microsoft/vscode/issues/100688 [2] https://spec.json5.org/

Unfortunately, JSON5 says keys can be ES5 IdentifierName[1]s, which means you must carry around Unicode tables. This makes it a non-option for small devices, for example. (I mean, not really, you technically could fit the necessary data and code in low single-digit kilobytes, but it feels stupid that you have to. Or you could just not do that but then it’s no longer JSON5 and what was the point of having a spec again?)

[1] https://es5.github.io/x7.html#x7.6

Re: Building a high performance JSON parser

#105

Post by mhh__ »

diarrhea wrote: post #26
vjerancrnjak wrote: post #9

Earlier quoted context omitted.

If your JSON always looks the same you can also do better than general JSON parsers.

I wonder: can fast, special-case JSON parsers be dynamically autogenerated from JSON Schemas? Perhaps some macro-ridden Rust monstrosity that spits out specialised parsers at compile time, dynamically…

It's relatively common in D application to use the compile time capabilities to generator a parser at compile time

Re: Building a high performance JSON parser

#106

Post by lgas »

coldtea wrote: post #82
jchw wrote: post #72

Looks pretty good! Even though I've written far too many JSON parsers already in my career, it's really nice to have a reference for how to think about making a reasonable, fast JSON parser, going through each step individually. That said, I will say one thing: you don't really need to have an explicit tokenizer for JSON. You can get rid of the concept of tokens and integrate parsing and tokenization entirely . This…

What line of work are you in that you've "written far too many JSON parsers already" in your career?!!!

Someone misunderstood the JSONParserFactory somewhere along the line.

Re: Building a high performance JSON parser

#107

Post by Maxion »

dh2022 wrote: post #73
isuckatcoding wrote: post #59

Earlier quoted context omitted.

Yes but for applications where you need to do ETL style transformations on large datasets, streaming is an immensely useful strategy. Sure you could argue go isn’t the right tool for the job but I don’t see why it can’t be done with the right optimizations like this effort.

If performance is important why would you keep large datasets in JSON format?

Because you work at or for some bureaucratic MegaCorp, that does weird things with no real logic behind it other than clueless Dilbert managers making decisions based on LinkedIn blogs. Alternatively desperate IT consultants trying to get something to work with too low of a budget and/or no access to do things the right way.

Be glad you have JSON to parse, and not EDI, some custom deliminated data format (with no or old documentation) - or shudders you work in the airline industry with SABRE.

Re: Building a high performance JSON parser

#108

Post by atombender »

minhazm wrote: post #28
diarrhea wrote: post #26

Earlier quoted context omitted.

I wonder: can fast, special-case JSON parsers be dynamically autogenerated from JSON Schemas? Perhaps some macro-ridden Rust monstrosity that spits out specialised parsers at compile time, dynamically…

For json schema specifically there are some tools like go-jsonschema[1] but I've never used them personally. But you can use something like ffjson[2] in go to generate a static serialize/deserialize function based on a struct definition. [1] https://github.com/omissis/go-jsonschema [2] https://github.com/pquerna/ffjson

Hey, go-jsonschema is my project. (Someone else just took over maintaining it, though.) It still relies on the standard Go parser; all it does it generate structs with the right types and tags.

Re: Building a high performance JSON parser

#109

Post by atombender »

dleeftink wrote: post #58
diarrhea wrote: post #26

Earlier quoted context omitted.

I wonder: can fast, special-case JSON parsers be dynamically autogenerated from JSON Schemas? Perhaps some macro-ridden Rust monstrosity that spits out specialised parsers at compile time, dynamically…

Somewhat tangentially related, Fabian Iwand posted this regex prefix tree visualiser/generator last week [0], which may offer some inspiration for prototyping auto generated schemas.

You forgot to include the link?

Re: Building a high performance JSON parser

#110

Post by mgaunard »

cbarrick wrote: post #98
mgaunard wrote: post #89

Earlier quoted context omitted.

You can in any systems programming language. Go is mostly a toy language for cloud people.

> toy language You may be surprised to hear that Go is used in a ton of large scale critical systems.

I don't consider cloud technology a critical system.
Post reply on HN