Write More Classes

> You can do the same with python's msgpack as the C# version

You can't. The unpacker in msgpack for Python only reads full objects. The C# version lets me go into and out of streaming at will and even skip past objects without reading them into memory. The only thing the Python version of msgpack can do with the unpacker is buffering up bytes internally until an object is ready. That object however could still expand into a 10GB blob of data which I would then have to postprocess.

> So, I guess the question I ask as someone who doesn't write a ton of python code is: Is that true?

I don't think it is. His objections to the json library API can be addressed without introducing a single class.

He seems to want access to the internal implementation of converting bytes into a single Python object. That is probably just an internal function that can just be exposed. If he wants to override it from the top level, the library could permit that by just adding the function as a default parameter.

He wants it to deal with streaming. Python generators can help with this. The library could be refactored into implementing a json.load_stream function instead, which takes a string generator and returns some kind of object stream generator. The original json.loads function could be replaced with a wrapper around it for backwards compatibility and for API users who don't want the additional complexity.

None of this involves writing classes. A generator could be seen as a type of class instance, and one can implement a generator with classes, but generators have a well known and simple API which eliminates the need to define a new API, as the traditional writing of classes would do.

It seems to me that the writer has little experience of not using classes, so wants to solve every problem he has with a class. The API problems he want to address are genuine, but he's blind to any solution that doesn't use classes.

I think there is an aspect to this article that won't be understood unless you're really part of the Python community. For a while now, one of Python's selling points to users from other languages has been "you don't need to write all that code" (most likely directed at Java), more specifically you can simply use a function rather than having to define a class just to define static methods etc. Over time, this has grown into the mantra of "You don't need a class just use a function/module". A lot of people seem to follow this viewpoint blindly, so much so that they consider it to be "Pythonic"; the way you "should" write python.

I think Armin's post is somewhat in line with the following post from the google testing blog. http://googletesting.blogspot.com/2008/12/static-methods-are...

> You would need another function that builds objects out of them.

I.e. you don't bundle your serialiser with your json parser. I think that's a good idea. How do you customise? most likely a callback that builds an object or returns the json fragment unmodified if it can't. It can be streamed too in order to accumulate / rebuild fragments of the tree.

Whether object is simpler here (builder / factory style), or another generator, that's a matter of taste mostly.

> For instance if you want to skip past objects.

You can't skip past objects at a tokenizer level, unless you implement logic of skipping whole structure. But that's what the parser does. Why don't you just skip the objects based on the streaming API? You don't have to construct them first - it's up to your deserialiser implementation if you want to ignore parts of the structure.

> Try using a streaming JSON API like Twitter's firehose.

From the documentation (unless I'm reading the wrong format description):

"""The body of a streaming API response consists of a series of newline-delimited messages, where "newline" is considered to be \r\n (in hex, 0x0D 0x0A) and "message" is a JSON encoded data structure or a blank line."""

That's not a huge JSON document. They use the newline delimiter as you described later. I don't think that's because "most libraries are horrible or useless for parsing streamed JSON". Why would you ever want to stream JSON which is an infinite and never complete object? What's wrong about splitting by newline? By making newlines the message separators, you'll never have to worry that your stream becomes broken due to some parsing error: bad message? ignore it skip to newline, continue with next one.

You make it sound like it's some kind of popularity contest, where people not choosing the language means it 'loses'. I disagree pretty heavily there - your hammer doesn't lose if you use your saw. The language's user loses if he is such a fanboy of hammers he refuses to use the saw. If Java improves to become the best tool (not saying it has/will), then choosing not use use Java because you 'moved on', is a failure on your part, not the tool.