Don't Pickle Your Data

"Use pickle with caution" would be better advice. There are plenty of cases where your data is under control and pickle is a huge time saver. It takes work to translate most data structures to/from json.

Re: Don't Pickle Your Data

#7

Post by falcolas » Wed, Feb 12, 2014, 9:10 PM UTC

JSON, while a fantastic lightweight data format, is insufficient for storing even moderately complex objects. It can not properly differentiate between tuples and lists, it can only accept strings for object keys, and it only stores unicode strings.

You also have to write custom (de)serializers if you want to store datetime objects (my personal pet peeve), any of the special python containers... basically any time you want anything other than an object, array, string or number.

Unless I'm dealing with untrusted data sources, or need to interoperate with other languages, I will keep using Pickle.

Re: Don't Pickle Your Data

#8

Post by jboynyc » Wed, Feb 12, 2014, 9:11 PM UTC

I like y_serial, a library to store compressed Python objects in a sqlite database.

http://yserial.sf.net/

Re: Don't Pickle Your Data

#9

Post by jfthiigsegbje » Wed, Feb 12, 2014, 9:19 PM UTC

jlebar wrote: post #3

This benchmark doesn't specify a pickle protocol, which forces Python to use a big, inefficient format. I filed a bug. https://github.com/benfred/bens-blog-code/issues/1

This is a huge thing he missed. Anyone using pickle for non-trivial work is doing cPickle with PROTOCOL_HIGHEST.

Re: Don't Pickle Your Data

#10

Post by [deleted] » Wed, Feb 12, 2014, 9:24 PM UTC

falcolas wrote: post #7

JSON, while a fantastic lightweight data format, is insufficient for storing even moderately complex objects. It can not properly differentiate between tuples and lists, it can only accept strings for object keys, and it only stores unicode strings. You also have to write custom (de)serializers if you want to store datetime objects (my personal pet peeve), any of the special python containers... basically any time yo…

[deleted]