Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't Pickle Your Data

benfrederickson.com

Post reply on HN
Print view View original

11–20 of 26 posts

Re: Don't Pickle Your Data

#11

Post by falcolas »

jlebar wrote: post #3

This benchmark doesn't specify a pickle protocol, which forces Python to use a big, inefficient format. I filed a bug. https://github.com/benfred/bens-blog-code/issues/1

I also went and tested it.

Baseline JSON:

    --------------------------------------------------------------------------------
    JSON
    packTime 0.612724065781 s -  163205.601975 items/s
    unpackTime 0.782174110413 s -  127848.772631 items/s
    size 174.26637
Baseline cPickle, ascii protocol:

    --------------------------------------------------------------------------------
    cPickle
    packTime 2.41442704201 s -  41417.6938297 items/s
    unpackTime 0.875658035278 s -  114199.831408 items/s
    size 286.26637
cPickle, highest protocol:

    --------------------------------------------------------------------------------
    cPickleHP
    packTime 1.02942800522 s -  97141.3245929 items/s
    unpackTime 0.583297967911 s -  171438.965162 items/s
    size 198.26637
For giggles, I evened the playing field and let cPickle at the same data structure as JSON, ascii protocol and highest protocol:

    --------------------------------------------------------------------------------
    cPickleJsonData
    packTime 0.642832040787 s -  155561.629874 items/s
    unpackTime 0.478959083557 s -  208786.101847 items/s
    size 205.53356


    --------------------------------------------------------------------------------
    cPickleHPjsonData
    packTime 0.285845041275 s -  349839.897708 items/s
    unpackTime 0.340456962585 s -  293722.881273 items/s
    size 175.26637
My conclusions? Serializing dictionaries is easier than serializing objects, and using a non-ascii protocol offers some obvious benefits. JSON is not obviously better than cPickle when comparing apples to apples.

It's also worth noting that comparing the json module to the pure python Pickle module isn't a fair fight either; json is at least partially written in C.

Re: Don't Pickle Your Data

#12

Post by boronine »

falcolas wrote: post #7

JSON, while a fantastic lightweight data format, is insufficient for storing even moderately complex objects. It can not properly differentiate between tuples and lists, it can only accept strings for object keys, and it only stores unicode strings. You also have to write custom (de)serializers if you want to store datetime objects (my personal pet peeve), any of the special python containers... basically any time yo…

I built an extensible type system on top of JSON for serialization and validation. Among other types, it ships with DateTime (it represents dates as ISO strings). It also lets you define your own types, as simple or complex as you need them to be. If you've got a minute to take a look, I would really appreciate some feedback :)

http://www.cosmic-api.com/docs/teleport/python/latest/

Re: Don't Pickle Your Data

#13

Post by paulgb »

falcolas wrote: post #11
jlebar wrote: post #3

This benchmark doesn't specify a pickle protocol, which forces Python to use a big, inefficient format. I filed a bug. https://github.com/benfred/bens-blog-code/issues/1

I also went and tested it. Baseline JSON: -------------------------------------------------------------------------------- JSON packTime 0.612724065781 s - 163205.601975 items/s unpackTime 0.782174110413 s - 127848.772631 items/s size 174.26637 Baseline cPickle, ascii protocol: -------------------------------------------------------------------------------- cPickle packTime 2.41442704201 s - 41417.6938297 items/s unp…

I wonder how ujson (pure-C) compares? https://pypi.python.org/pypi/ujson

Re: Don't Pickle Your Data

#14

Post by thu »

At the end of the post he says that only Python can parse Pickle. This is not really true. You can write a Pickle parsing library in any language (maybe some special use would not be possible).

For instance I have implemented enough of it[0] to use in a drop-in replacement for Graphite[1].

[0]: https://github.com/noteed/python-pickle [1]: http://graphite.wikidot.com/

Re: Don't Pickle Your Data

#15

Post by drdaeman »

falcolas wrote: post #7

JSON, while a fantastic lightweight data format, is insufficient for storing even moderately complex objects. It can not properly differentiate between tuples and lists, it can only accept strings for object keys, and it only stores unicode strings. You also have to write custom (de)serializers if you want to store datetime objects (my personal pet peeve), any of the special python containers... basically any time yo…

Heh. Pickle is not really sufficient for some edge cases, too. Luckily, there's Dill that can take almost perfect snapshots of the whole interpreter: https://pypi.python.org/pypi/dill

Re: Don't Pickle Your Data

#16

Post by vbit »

Your benchmarks don't mean much because you're giving different data to different packers.

cPickle gets a list of objects but json gets a list of dictionaries? The cost of converting the objects into the dictionaries is conveniently excluded from the json benchmark.

You should try serializing the same list of dictionaries, use the highest pickle protocol and repost results.

Re: Don't Pickle Your Data

#19

Post by silveira » 

  >>> import json
  >>> json.dumps(set())
  Traceback (most recent call last):
    File "", line 1, in 
    File "/usr/lib/python2.7/json/__init__.py", line 231, in dumps
      return _default_encoder.encode(obj)
    File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
      chunks = self.iterencode(o, _one_shot=True)
    File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
      return _iterencode(o, 0)
    File "/usr/lib/python2.7/json/encoder.py", line 178, in default
      raise TypeError(repr(o) + " is not JSON serializable")
  TypeError: set([]) is not JSON serializable
  >>> from decimal import Decimal
  >>> json.dumps(Decimal(1))
  Traceback (most recent call last):
    File "", line 1, in 
    File "/usr/lib/python2.7/json/__init__.py", line 231, in dumps
      return _default_encoder.encode(obj)
    File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
      chunks = self.iterencode(o, _one_shot=True)
    File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
      return _iterencode(o, 0)
    File "/usr/lib/python2.7/json/encoder.py", line 178, in default
      raise TypeError(repr(o) + " is not JSON serializable")
  TypeError: Decimal('1') is not JSON serializable

Re: Don't Pickle Your Data

#20

Post by JasonFruit »

silveira wrote: post #19

>>> import json >>> json.dumps(set()) Traceback (most recent call last): File " ", line 1, in File "/usr/lib/python2.7/json/__init__.py", line 231, in dumps return _default_encoder.encode(obj) File "/usr/lib/python2.7/json/encoder.py", line 201, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode return _iterencode(o, 0) File "/usr/lib/python2.7/jso…

What, precisely, is your point? The article points out JSON's limitations.
Post reply on HN