Live data from Hacker News

Don't Pickle Your Data

benfrederickson.com

21–26 of 26 posts

Re: Don't Pickle Your Data

#21
post #7

JSON, while a fantastic lightweight data format, is insufficient for storing even moderately complex objects. It can not properly differentiate between tuples and lists, it can only accept strings for object keys, and it only stores unicode strings. You also have to write custom (de)serializers if you want to store datetime objects (my personal pet peeve), any of the special python containers... basically any time yo…

Also, most critically, JSON doesn't allow circular references.

Re: Don't Pickle Your Data

#22
post #7

JSON, while a fantastic lightweight data format, is insufficient for storing even moderately complex objects. It can not properly differentiate between tuples and lists, it can only accept strings for object keys, and it only stores unicode strings. You also have to write custom (de)serializers if you want to store datetime objects (my personal pet peeve), any of the special python containers... basically any time yo…

Heh. Pickle is not really sufficient for some edge cases, too. Luckily, there's Dill that can take almost perfect snapshots of the whole interpreter: https://pypi.python.org/pypi/dill

Also Dill's developer Michael Mckerns is super responsive and helpful. Hopefully someday pickle will be replaced / augmented with dill in the standard distribution.

Re: Don't Pickle Your Data

#23
> Given the downsides though, its worth writing the little bit of code necessary to convert your objects to a JSON-able form if your code is ever going to be used by people other than yourself.

Disagree strongly. You have no idea how complex my graphs of (pretty damn complex) python objects are and how much I value being able to change their organization without having to rewrite the serialization code every time I do so.

Don't fear pickle. Just don't expect it to be secure or amazingly fast.

Re: Don't Pickle Your Data

#24
post #5

In case anyone doesn't know, Python has had a json library since version 2.6.

That's not the issue. The issue is that python objects are not serializable into json without writing a custom serializer. Rather than telling devs not to use pickle, why not build a json serializer that can handle simple objects and try to get it adopted into the python standard library?

Re: Don't Pickle Your Data

#25
post #4

I thought pickle output was just JSON? Shows what I know.

If it was JSON, how would you pickle dictionaries with tuples as keys?

The same way you would in JavaScript: turn it into a string first. "(1, 2)" Its not that serializing python objects to json is impossible, just that there is not a straightforward implementation that everyone agrees is correct.

For a fully general python json serializer/deserializer, you'd need a bunch of extra annotations etc that would make it look very weird compared with what most people expect when they think of json.

Post reply on HN