Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Json vs. simplejson vs. ujson

jyotiska.github.io

Post reply on HN
Print view View original

51–60 of 75 posts

Re: Json vs. simplejson vs. ujson

#51

Post by MagicWishMonkey »

chojeen wrote: post #48

Maybe this is a dumb question, but is json (de)serialization really a bottleneck for python web apps in the real world?

Depends on the app. My previous job required processing thousands of address book contact records uploaded to the server in a massive list. It was not unsual for some of these objects to exceed 10mb (when serialized to disk).

The default json module took close to 5 seconds to deserialize the payload once it hit the server, while ujson could do the same work in a fraction of the time (less than a second). 5 seconds might not seem like a whole lot when the import process as a whole could take 30 seconds or so, but when the user is stuck staring at their device it makes sense to cut down the response time any way you can.

Re: Json vs. simplejson vs. ujson

#52

Post by kelseyfrancis »

michaelmior wrote: post #37

> ultrajson ... will not work for un-serializable collections So I can't serialize things with ultrajson that aren't serializable? I must be missing something in this statement. > The verdict is pretty clear. Use simplejson instead of stock json in any case... The verdict seems clear (based solely on the data in the post) that ultrajson is the winner.

> The verdict seems clear (based solely on the data in the post) that ultrajson is the winner.

ultrajson isn't a drop-in replacement, though, because it doesn't support sort_keys.

Re: Json vs. simplejson vs. ujson

#53

Post by est »

chojeen wrote: post #48

Maybe this is a dumb question, but is json (de)serialization really a bottleneck for python web apps in the real world?

For some really large JSONs out there in the wild, yes, it's a big bottleneck.

We ended up using ijson.

Re: Json vs. simplejson vs. ujson

#54

Post by twic »

jbergstroem wrote: post #27

I just want to add another library in here which – at least in my world – is replacing json as the number one configuration and serialisation format. It's called libucl and it's main consumer is probably the new package tool in FreeBSD: `pkg` Its syntax is nginx-like but can also parse strict json. It's pretty fast too. More info here: https://github.com/vstakhov/libucl

The automatic array creation feature [1] seems misguided. It means that as a programmer consuming a configuration file, i can't know whether a given field will be a scalar or an array. I recently worked on a JavaScript API that had that behaviour, and it was a pain.

Apart from that, though, this looks like a really good format.

[1] https://github.com/vstakhov/libucl#automatic-arrays-creation

Re: Json vs. simplejson vs. ujson

#55

Post by jarman »

rat87 wrote: post #30
JoshTriplett wrote: post #20

Earlier quoted context omitted.

Seems like there should be a standard Python mechanism for constructing "atoms" or "symbols" that automatically get commoned up.

I'm pretty sure symbols are not meant to be created from "user" input where user is untrusted, can't this lead to ddos atacks? Same thing for interning. De-Duping doesn't have that risk.

Depends on symbol implementations and intended usage.

For example Erlang symbols are deeply ingrained into language, and vm doesn't even garbage collects them, so creating symbols from user data is basically giving user 'crush vm' button.

On the other hand, if symbols are treated as another data type, as string with some optimizations - no such problems shall arise

Re: Json vs. simplejson vs. ujson

#56

Post by js2 »

I'll have to try ultrajson for my use case, but when I benchmarked pickle, simplejson and msgpack, msgpack came out the fastest. I also tried combining all three formats with gzip, but that did not help. Primarily I care about speed when deserializing from disk.

Re: Json vs. simplejson vs. ujson

#57

Post by TazeTSchnitzel »

borman wrote: post #44

The problem with all (widely known) the non-standard JSON packages is, they all have their gotchas. cjson's way of handling unicode is just plain wrong: it uses utf-8 bytes as unicode code points. ujson cannot handle large numbers (somewhat larger than 2 63, i've seen a service that encodes unsigned 64-bit hash values in JSON this way: ujson fails to parse its payloads). With simplejson (when using speedups module),…

There are so many poorly-written JSON decoders out there. I've had the misfortune of fixing two of PHP's to follow JSON's case-sensitivity and whitspace rules properly.

Re: Json vs. simplejson vs. ujson

#58

Post by akoumjian »

We took a look at ujson about a year ago and found that it failed loading even json structures that went 3 layers deep. I also recall issues handling unicode data.

It was a big disappointment after seeing these kinds of performance improvements.

Re: Json vs. simplejson vs. ujson

#59

Post by michaelmior »

kelseyfrancis wrote: post #52
michaelmior wrote: post #37

> ultrajson ... will not work for un-serializable collections So I can't serialize things with ultrajson that aren't serializable? I must be missing something in this statement. > The verdict is pretty clear. Use simplejson instead of stock json in any case... The verdict seems clear (based solely on the data in the post) that ultrajson is the winner.

> The verdict seems clear (based solely on the data in the post) that ultrajson is the winner. ultrajson isn't a drop-in replacement, though, because it doesn't support sort_keys.

Fair enough. Although I'm not sure why one would want that behaviour given that there is no guarantee of ordering when a particular JSON file is processed with any other library.

Re: Json vs. simplejson vs. ujson

#60

Post by TillE »

crdoconnor wrote: post #47
jbergstroem wrote: post #27

I just want to add another library in here which – at least in my world – is replacing json as the number one configuration and serialisation format. It's called libucl and it's main consumer is probably the new package tool in FreeBSD: `pkg` Its syntax is nginx-like but can also parse strict json. It's pretty fast too. More info here: https://github.com/vstakhov/libucl

Any particular reason to use this over YAML for configuration?

Macros seem to be the key distinguishing feature. It's a good idea, sort of borrowing from template engines.
Post reply on HN