Json vs. simplejson vs. ujson • Hacker News Board

Re: Json vs. simplejson vs. ujson

#21

Post by tlb » Mon, Apr 06, 2015, 8:05 AM UTC

JoshTriplett wrote: post #20

jmoiron wrote: post #7

When I wrote the same kind of article in Nov 2011 [1], I came to similar conculsions; ujson was blowing everyone away. However, after swapping a fairly large and json-intensive production spider over to ujson, we noticed a large increase in memory use. When I investigated, I discovered that simplejson reused allocated string objects, so when parsing/loading you basically got string compression for repeated string key…

Seems like there should be a standard Python mechanism for constructing "atoms" or "symbols" that automatically get commoned up.

Check out intern(): https://docs.python.org/2/library/functions.html#intern

Re: Json vs. simplejson vs. ujson

#22

Post by gpvos » Mon, Apr 06, 2015, 8:42 AM UTC

azinman2 wrote: post #16

wodenokoto wrote: post #11

How hard is it to draw a bar graph? I'd imagine it is easier than creating an ASCII table and then turning that into an image, but I've never experimented with the latter.

Not helpful.

Actually sounds like an honest question to me.

Re: Json vs. simplejson vs. ujson

#23

Post by willvarfar » Mon, Apr 06, 2015, 8:45 AM UTC

(My own due diligence when working with serialisation: http://stackoverflow.com/questions/9884080/fastest-packing-o...

I leave this here in case it helps others.

We had other focus such as good for both python and java.

At the time we went msgpack. As msgpack is doing much the same work as json, it just shows that the magic is in the code not the format..)

Re: Json vs. simplejson vs. ujson

#24

Post by jkire » Mon, Apr 06, 2015, 8:53 AM UTC

> We have a dictionary with 3 keys

What about larger dictionaries? With such a small one I would be worried that a significant proportion of the time would be simple overhead.

[Warning: Anecdote] When we were testing out the various JSON libraries we found simplejson much faster than json for dumps. We used large dictionaries.

Was the simplejson package using its optimized C library?

Re: Json vs. simplejson vs. ujson

#25

Post by jyotiska » Mon, Apr 06, 2015, 9:07 AM UTC

wodenokoto wrote: post #11

How hard is it to draw a bar graph? I'd imagine it is easier than creating an ASCII table and then turning that into an image, but I've never experimented with the latter.

You are right. I had to create the ASCII tables, because I was not able to draw tables on Medium.

Re: Json vs. simplejson vs. ujson

#26

Post by jkire » Mon, Apr 06, 2015, 9:11 AM UTC

jkire wrote: post #24

> We have a dictionary with 3 keys What about larger dictionaries? With such a small one I would be worried that a significant proportion of the time would be simple overhead. [Warning: Anecdote] When we were testing out the various JSON libraries we found simplejson much faster than json for dumps. We used large dictionaries. Was the simplejson package using its optimized C library?

> In this experiment, we have stored all the dictionaries in a list and dumped the list using json.dumps()

I completely failed to read this the first time I went through. I guess this is equivalent to dumping bigger dictionaries.

> [Warning: Anecdote] When we were testing out the various JSON libraries we found simplejson much faster than json for dumps.

Turns out we were using sort_keys=True option, which apparently makes simplejson much faster than json.

Re: Json vs. simplejson vs. ujson

#27

Post by jbergstroem » Mon, Apr 06, 2015, 9:19 AM UTC

I just want to add another library in here which – at least in my world – is replacing json as the number one configuration and serialisation format. It's called libucl and it's main consumer is probably the new package tool in FreeBSD: `pkg`

Its syntax is nginx-like but can also parse strict json. It's pretty fast too.

More info here: https://github.com/vstakhov/libucl

Re: Json vs. simplejson vs. ujson

#28

Post by stock_toaster » Mon, Apr 06, 2015, 9:23 AM UTC

jbergstroem wrote: post #27

I just want to add another library in here which – at least in my world – is replacing json as the number one configuration and serialisation format. It's called libucl and it's main consumer is probably the new package tool in FreeBSD: `pkg` Its syntax is nginx-like but can also parse strict json. It's pretty fast too. More info here: https://github.com/vstakhov/libucl

At one point I looked into using it, but there werent any python bindings at the time, and I didnt have the time (for the project) to write any. Are there any good language libs for it these days?

Re: Json vs. simplejson vs. ujson

#29

Post by JoshTriplett » Mon, Apr 06, 2015, 9:33 AM UTC

tlb wrote: post #21

JoshTriplett wrote: post #20

Earlier quoted context omitted.

Seems like there should be a standard Python mechanism for constructing "atoms" or "symbols" that automatically get commoned up.

Check out intern(): https://docs.python.org/2/library/functions.html#intern

Appears to be deprecated though.

Re: Json vs. simplejson vs. ujson

#30

Post by rat87 » Mon, Apr 06, 2015, 9:37 AM UTC

JoshTriplett wrote: post #20

jmoiron wrote: post #7

When I wrote the same kind of article in Nov 2011 [1], I came to similar conculsions; ujson was blowing everyone away. However, after swapping a fairly large and json-intensive production spider over to ujson, we noticed a large increase in memory use. When I investigated, I discovered that simplejson reused allocated string objects, so when parsing/loading you basically got string compression for repeated string key…

Seems like there should be a standard Python mechanism for constructing "atoms" or "symbols" that automatically get commoned up.

I'm pretty sure symbols are not meant to be created from "user" input where user is untrusted, can't this lead to ddos atacks? Same thing for interning. De-Duping doesn't have that risk.