Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Json vs. simplejson vs. ujson

jyotiska.github.io

Post reply on HN
Print view View original

61–70 of 75 posts

Re: Json vs. simplejson vs. ujson

#61

Post by munificent »

rat87 wrote: post #30
JoshTriplett wrote: post #20

Earlier quoted context omitted.

Seems like there should be a standard Python mechanism for constructing "atoms" or "symbols" that automatically get commoned up.

I'm pretty sure symbols are not meant to be created from "user" input where user is untrusted, can't this lead to ddos atacks? Same thing for interning. De-Duping doesn't have that risk.

Lua has an interesting approach here. In Lua, all strings are interned. If you have "two" strings that consist of the same bytes, you are guaranteed that they have the same address and are the same object. Basically, every time a string is created from some operation, it's looked up in a hash table of the existing strings and if an identical one is found, that gets reused.

However, that hash table stores weak references to those strings. If nothing else refers to a string, the GC can and will remove it from the string table.

This gives you great memory use for strings and optimally fast string comparisons. The cost is that creating a string is probably a bit slower because you have to check the string table for the existing one first.

It's an interesting set of trade-offs. I think it makes a lot of sense for Lua which uses hash tables for everything, including method dispatch and where string comparison must be fast. I'm not sure how much sense it would make for other languages.

Re: Json vs. simplejson vs. ujson

#62

Post by michaelmior »

rat87 wrote: post #30
JoshTriplett wrote: post #20

Earlier quoted context omitted.

Seems like there should be a standard Python mechanism for constructing "atoms" or "symbols" that automatically get commoned up.

I'm pretty sure symbols are not meant to be created from "user" input where user is untrusted, can't this lead to ddos atacks? Same thing for interning. De-Duping doesn't have that risk.

I think most JSON structures are unlikely to have user input be used as keys. This is also likely where there would be the most benefit from interning since keys are often repeated many times.

Re: Json vs. simplejson vs. ujson

#63

Post by apendleton »

michaelmior wrote: post #37

> ultrajson ... will not work for un-serializable collections So I can't serialize things with ultrajson that aren't serializable? I must be missing something in this statement. > The verdict is pretty clear. Use simplejson instead of stock json in any case... The verdict seems clear (based solely on the data in the post) that ultrajson is the winner.

> So I can't serialize things with ultrajson that aren't serializable? I must be missing something in this statement.

This might not be what they're talking about, but I did run into what might be the same issue when looking at ujson before. The builtin JSON module lets you define custom serializations for types that aren't natively JSON-serializable; we had an application that did that with datetime objects, encoding them as ISO 8601 date strings. ujson doesn't support anything like that; you have to make sure everything is one of the JSON types already before encoding.

Re: Json vs. simplejson vs. ujson

#64

Post by thethimble »

jyotiska wrote: post #14
jmoiron wrote: post #7

When I wrote the same kind of article in Nov 2011 [1], I came to similar conculsions; ujson was blowing everyone away. However, after swapping a fairly large and json-intensive production spider over to ujson, we noticed a large increase in memory use. When I investigated, I discovered that simplejson reused allocated string objects, so when parsing/loading you basically got string compression for repeated string key…

Hey Jason. Thats pretty interesting. I have also noticed similar things but for my case, we needed faster loading/unloading for some cases, hence ujson.

I would love to see benchmarks in PyPy as well! I wonder how well a JIT would handle de/serialization.

Re: Json vs. simplejson vs. ujson

#66

Post by smerritt »

borman wrote: post #44

The problem with all (widely known) the non-standard JSON packages is, they all have their gotchas. cjson's way of handling unicode is just plain wrong: it uses utf-8 bytes as unicode code points. ujson cannot handle large numbers (somewhat larger than 2 63, i've seen a service that encodes unsigned 64-bit hash values in JSON this way: ujson fails to parse its payloads). With simplejson (when using speedups module),…

Agreed, especially about simplejson. I work on a project that uses simplejson, and it leads to ugly type checking all over the place because you never know what your JSON string got turned into. For example:

https://github.com/openstack/swift/blob/39c1362a4f5a7df75730...

and https://github.com/openstack/swift/blob/39c1362a4f5a7df75730...

and https://github.com/openstack/swift/blob/39c1362a4f5a7df75730...

and many more just like those.

The worst part is the bugs that appear or disappear depending on whether simplejson's speedups module is in use or not.

Re: Json vs. simplejson vs. ujson

#68

Post by mdaniel »

michaelmior wrote: post #59
kelseyfrancis wrote: post #52

Earlier quoted context omitted.

> The verdict seems clear (based solely on the data in the post) that ultrajson is the winner. ultrajson isn't a drop-in replacement, though, because it doesn't support sort_keys.

Fair enough. Although I'm not sure why one would want that behaviour given that there is no guarantee of ordering when a particular JSON file is processed with any other library.

I don't know what they do with it, but it's handy for writing tests against an expected JSON file: assert json.dumps(expected, sort_keys=True) == json.dumps(obj, sort_keys=True) # where expected was json.load()-ed and obj was produced by the function

Re: Json vs. simplejson vs. ujson

#70

Post by metaphorm »

chojeen wrote: post #48

Maybe this is a dumb question, but is json (de)serialization really a bottleneck for python web apps in the real world?

depends on what you're doing.

for the typical AJAX call for some rows of data selected from a datastore and JSON encoded, then no the JSON encoding is not the bottleneck, the network latency and database io time dominate the time it takes to JSON encode the data.

however, consider an alternative kind of task that might, for example, produce a big JSON dump of thousands of records. this is fairly typical of a data export of some kind. the network and database time for this request is the same as for the smaller one, but now instead of JSON encoding 50 records you're encoding 50000 records. it can start to add up. a poorly optimized JSON library will add multiple full seconds to your response time here.

Post reply on HN