The cost of parsing JSON • Hacker News Board

Re: The cost of parsing JSON

#281

Post by manfredo » Thu, Sep 19, 2019, 4:40 AM UTC

kllrnohj wrote: post #275

manfredo wrote: post #261

Earlier quoted context omitted.

What is the "completely different thing" you're referring to here. Between: 1. Having a static JSON string, and decoding that string. and 2. having a static blob, and using protobufs to decode that blob. these two things accomplish the same thing. I'm not sure why you seem to think one is a "cold start" and the other is "hot" - they're both "single, one-time object creation". The former is going to be parsing ints an…

Are you including the cost of loading protobuf itself? You seem to be basing your argument on an assumed already present & loaded protobuf library. You need to benchmark starting from nothing at all. Your link that you seem to be basing this off of has a loaded and fully JIT'd protobuf. That's not the start state.

You can measure the impact on loading time, and the size of the protobuf implementation you're using probably has an impact on the threshold at which it becomes more efficient. I don't doubt that parsing a 500 character long JSON string is probably faster than loading a protobuf to do it instead. In fact, apparently this JSON parsing trick is only effective beyond 10K or so. But past a certain threshold memory bandwidth is more crucial than loading code. If your data consists mostly of booleans and integers then JSON can often be an order of magnitude larger in size than protobufs. If it's compressed, then decompressing it takes clock cycles and the parsing code is still parsing the larger uncompressed JSON text. A protobuf library can often skip compression altogether by virtue of using normal ints and bits for numbers and booleans. So while the protobuf library does have some additional overhead it's often higher throughput for many types of data.

Re: The cost of parsing JSON

#282

Post by dmix » Thu, Sep 19, 2019, 5:15 AM UTC

Klathmon wrote: post #81

nostrebored wrote: post #72

Earlier quoted context omitted.

Why not just use promises? side note: legit question, I don't do web/app dev

Because JSON.parse blocks the thread it's in, and JS is single threaded [1]. So even if you put it behind a promise, when that promise actually runs, it will block the thread. In essence, using promises (or callbacks or timeouts or anything else like that) allows you to delay the thread-blocking, but once the code hits `JSON.parse`, no other javascript will run until it completes. And since no other javascript will r…

I'm guessing with Oboe.js you solved this by capturing a stream(?) of JSON but only parsing relevant chunks as they appear and match the selector? Or do you simply load the larger chunks at once (either by a request or embedding JSON into the template server side) instead of streaming?

http://oboejs.com/examples#demarshalling-json-to-an-oop-mode...

I could see the value in this for sure. I currently have a problem of loading a ton of JS for some users who have thousands of objects embedded in the view with Rails using toJSON() in a . It’s creating far too much weight on the frontend. I’ve been considering fetching it via a simple REST request instead.

Re: The cost of parsing JSON

#283

Post by IggleSniggle » Thu, Sep 19, 2019, 5:26 AM UTC

Sephr wrote: post #111

IggleSniggle wrote: post #12

Commenting the security issue from the end of explainer for visibility. I’m having flashbacks to the Java serialize vulnerabilities from a couple years ago. ECMAScript and JSON do not have the same set of escape characters: ``` Note: It’s crucially important to post-process user-controlled input to escape any special character sequences, depending on the context. In this particular case, we’re injecting into a tag, s…

Why would you ever be escaping HTML in client-side JS? You should be using appropriate DOM APIs (which don't include innerHTML) to manipulate the document.

I think you’ve missed the vulnerability. You can use appropriate DOM APIs all you want and be hit by a malicious escaper if you don’t serialize in the right way. At some point, your js is inserted into the document via a script tag. If you use the JSON parser to initialize your data more quickly, and your data has user input anywhere within it, then if you aren’t careful about encoding/decoding your string, the attacker’s can abruptly interrupt your own script simply by inserting `//attacker script here`. It is comparable to an SQL injection attack, but instead it is HTML injection that is made possible if you use JSON.stringify without taking the differing character specification into account (the example in the write up shows one good way to do this).

The attacker is escaping JSON.parse to escape HTML.

Re: The cost of parsing JSON

#284

Post by cellularmitosis » Thu, Sep 19, 2019, 6:27 AM UTC

t1amat wrote: post #237

adimitrov wrote: post #206

Earlier quoted context omitted.

> Part of me still thinks that he just wanted to learn/use the JNI and that project seemed like the perfect target. Lol. As a dev who sometimes goes off chasing wind mills, that's 99% of the reason why I do it. I find something nice to tinker with, and when my brain goes "ooh, shiny" I stop giving a shit about anyone's bottom line. To be fair, it usually turns out for the better for the project and its code base! But…

> Companies should be willing to take these kinds of informed risks in order to improve their employees' ability, and therefore the quality of their product. Perhaps they should be willing, but your description of this distraction does not including informing the Company and allowing them to determine whether it’s a risk they are willing to accept. You decided for them because you didn’t want to receive the answer “n…

> "This isn’t right."

I'm afraid the morality of this situation isn't so black-and-white.

In industry, there is always a tension between production and research: cranking out widgets vs. getting better at cranking out widgets.

A dev who spends 100% of their time cranking out widgets is stagnating. That's actually not what your employer wants, despite the fact that their agile process seems to imply that ticket cranking shall be the whole of your focus.

If you ask employers if they expect you to improve your skills over time, they would absolutely say "yes". But if you ask for permission to chase a specific white whale, you will hear "no". Everyone agrees they should be saving for the future, but "not this paycheck".

Taking the naive moral approach here and spending 100% of your time on tickets is not "what's right". If anything, that's you being taken advantage of by your employer -- sacrificing the advancement of your career in the name of short-term sprint velocity gains. On top of that, stagnation is not what your employer really wants anyway.

(edit: the above excludes companies which have explicit "20% time").

Re: The cost of parsing JSON

#285

Post by wsc981 » Thu, Sep 19, 2019, 6:53 AM UTC

Gibbon1 wrote: post #214

19ylram49 wrote: post #163

Earlier quoted context omitted.

To be honest, I understand the bit of backlash that I’ve received here and I think it’s well-deserved since I should’ve worded my statement better. Thank you for your comments. You all are correct re firing someone over mistakes and seemingly trivial matters. I was mostly referring to software engineers who make impactful decisions without good reason and/or without properly assessing the trade-offs. I think it’s fai…

> And this was a very brilliant software engineer, but he was consistently making poor decisions like this. That is something I've noticed. Brilliance doesn't go hand in hand with making prudent and wise decisions. > I did my best to shield that software engineer from them I've found rather painfully that you shouldn't shield guys like that when they go off on their own to make mistakes. Other thing, you have a team…

> That is something I've noticed. Brilliance doesn't go hand in hand with making prudent and wise decisions.

Reminds me that John's Carmack wife told Carmack that she wouldn't allow him to bankrupt the family with his space hobby company (Armadillo Aerospace) :)

Re: The cost of parsing JSON

#286

Post by dkersten » Thu, Sep 19, 2019, 8:20 AM UTC

krferriter wrote: post #258

dkersten wrote: post #170

Earlier quoted context omitted.

Its an uneven power dynamic though. The employers typically hold many more cards than the employees.

"typically"? I think you mean "always". This notion of it being an equal relationship in both directions is ridiculous.

Well, there are those rare cases where the company relies on a person or small group to continue their business, but it’s not a typical situation.

Re: The cost of parsing JSON

#287

Post by bestest » Thu, Sep 19, 2019, 9:11 AM UTC

Klathmon wrote: post #99

sbr464 wrote: post #97

Earlier quoted context omitted.

You can also pretty easily use a web worker now, they work well. Here's [1] an example with React hooks. Example fibonacci worker code that doesn't block the UI, even at larger calculations const fib = n => (n { console.log('fibonacci worker onmessage', msg) postMessage({ num: msg.data, result: fib(msg.data) }) } [1] https://github.com/bharathnayak03/react-webworker-hook

Web workers won't work in this case because they need to serialize all data going into and out of them (with the exception of TypedArrays). So passing a string to a worker and having it JSON.parse it works great. But when you go to pass that object back to the main thread, it implicitly does a JSON.stringify and a JSON.parse back on the main thread (technically it's called a "Structured Copy", but it's mostly the sam…

If this really NEEDS to be a client-side-only solution, I still believe a worker is the only way to go. Only, in this case, it needs to behave like an API. So your worker not only parses the JSON, but also responds to post messages with only the data that is requested from it.

Why? Because a large JSON structure is most probably just a large JSON structure, but you most probably don't need it as a whole. You may need a total count of items, you may need a paginated set of items, or only a certain item or a set of fields of items — well, an API.

Re: The cost of parsing JSON

#288

Post by adimitrov » Thu, Sep 19, 2019, 9:52 AM UTC

t1amat wrote: post #237

adimitrov wrote: post #206

Earlier quoted context omitted.

> Part of me still thinks that he just wanted to learn/use the JNI and that project seemed like the perfect target. Lol. As a dev who sometimes goes off chasing wind mills, that's 99% of the reason why I do it. I find something nice to tinker with, and when my brain goes "ooh, shiny" I stop giving a shit about anyone's bottom line. To be fair, it usually turns out for the better for the project and its code base! But…

> Companies should be willing to take these kinds of informed risks in order to improve their employees' ability, and therefore the quality of their product. Perhaps they should be willing, but your description of this distraction does not including informing the Company and allowing them to determine whether it’s a risk they are willing to accept. You decided for them because you didn’t want to receive the answer “n…

But it is an informed risk. I was hired to do work in 4 different languages, on the frontend and the backend, plus CSS and HTML. I get to weigh in on UX decisions, I get to design service infrastructure.

Was I born this way? No. I need this overhead, that's just part of being a dev (within reason).

If you require me to do lots of things, there's overhead. If you want a ticket drone for your Scrumfall projects, get a ticket drone.

Re: The cost of parsing JSON

#289

Post by hvidgaard » Thu, Sep 19, 2019, 9:54 AM UTC

mamon wrote: post #216

pushpop wrote: post #186

Earlier quoted context omitted.

Thankfully most other developed country’s healthcare system doesn’t penalise individuals quite so significantly as the broken system you Americans keep voting for. I’m not saying the UK or other European counties have the perfect healthcare systems either but at least we aren’t tied to a job we don’t like because losing our company’s health scheme is too scary to consider.

If you're not paying for it with your money, then you have to pay with your time: public healthcare systems, like those in Europe are known for the long wait times for patients requiring surgery, or other costly procedures. Also, traveling to the US for treatment is still a thing, because new, advanced treatments are developed and first implemented in the US, so all that money spent give you something in return.

From the moment my GP refers me to a hospital for whatever reason they need to look at me, they have 8 days to respond, and must have a diagnosis within 30 days. Treatment is usually not long after and almost always proportional to the situation.

If a potential life threatening disease is suspected diagnosis and treatment must have begone after no more than 2 weeks. Most of the time it's a matter of days. If the public hospitals cannot do that I'm free to go to private hospitals without paying anything.

How is that waiting a very long time?

Re: The cost of parsing JSON

#290

Post by smileypete » Thu, Sep 19, 2019, 9:56 AM UTC

nojvek wrote: post #277

Klathmon wrote: post #62

Earlier quoted context omitted.

I find this really interesting, because at some point the absolute performance benefits of `JSON.parse` is overshadowed by the fact that it blocks the main thread. I worked on an app a while ago which would have to parse 50mb+ JSON objects on mobile devices. In some cases (especially on mid-range and low-end devices) it would hang the main thread for a couple seconds! So I ended up using a library called oboe.js [1]…

If you really need to work with large data files 1mb+. Json is a terrible format. You should look into flat buffers. It’s like having indexed json where there is no parsing cost. You can have millions of rows and nested objects and it will only read the bytes it needs. It is length prefix encoded format so it’s pretty safe to work in a streaming manner too.

Good article on how Facebook used them for their mobile app:

https://code.fb.com/android/improving-facebook-s-performance...