Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Convert JSON to a Unix-friendly line-based format

github.com

Post reply on HN
Print view View original

11–20 of 22 posts

Re: Convert JSON to a Unix-friendly line-based format

#11

Post by jedsmith »

masklinn wrote: post #10
jedsmith wrote: post #9

import os.path as p Renaming imports to single letters bugs me. I went to see where it's used, and it isn't. Binding pyflakes to Cmd+S in TextMate was the best thing I ever did, and it would have caught this. pyflakes is seriously awesome in that role, and pylint before committing. I'm also surprised that simplejson is used, instead of the built-in json in Python 2.6 and up. A good solution is: try: import json excep…

> I'm also surprised that simplejson is used 2.6's json lacks simplejson's C accelerators [0] which makes it roughly 20 times slower than simplejson (2.7's simplejson is also ~50% slower than simplejson, as new performance optimizations were added in 2.1.0, as well as memoizations). Simplejson is also updated more often, which lead to it having less bugs, and interesting new features (the ability to natively serializ…

I elect to stick with the standard library in all cases, so I know how my software will perform everywhere without a redundant external dependency. Sticking with the standard library means you'll eventually get those improvements, too.

Re: Convert JSON to a Unix-friendly line-based format

#12

Post by jedsmith »

adulau wrote: post #5
haberman wrote: post #2

"Everyone I know prefers to work with JSON over XML, but sadly there is a sore lack of utilities of the quality or depth of html-xml-utils and XMLStarlet for actually processing JSON data in an automated fashion, short of writing an ad hoc processor in your favourite programming language." Actually there is just such a suite of utilites! See https://github.com/benbernard/RecordStream

RecordStream seems more complete but requires much more dependencies. I really like the simplicity of jsonpipe and how easy is to use it: curl -s "http://feeds.delicious.com/v2/json/adulau | jsonpipe -s "#" | grep "\#u" | cut -f2 | sed -e"s/\"//g" | xargs -d"\n" wget -r -l 1 –p –-convert-links A simple example to take a local mirror of my last del.icio.us bookmarks...

"simple"

Re: Convert JSON to a Unix-friendly line-based format

#13

Post by edd_dumbill »

Line-based processing is still important! This work reminds me of an article I wrote 11 years ago covering Sean McGrath's work on PYX—a line-based format for XML—see http://www.xml.com/pub/a/2000/03/15/feature/index.html.

That work derived from that of Charles Goldfarb on SGML, dating from 1989 on ESIS, ISO 8879.

We'll always be downsampling to something we can use with sed, grep and awk. They're too handy not to.

Re: Convert JSON to a Unix-friendly line-based format

#14

Post by sigil »

Interesting. For faster flattening of nested structures into paths -- and the inverse operation, unflattening -- you could use this Python C extension:

https://github.com/acg/python-flattery

Full disclosure, I'm the author. ;) It uses "." as the path separator, but would be easy to allow "/".

Re: Convert JSON to a Unix-friendly line-based format

#15

Post by sigil »

Maybe you're solving a different problem, but I'm not sure emitting one key-value pair per line is ultimately the way to go:

  $ echo '[{"a": [{"b": {"c": ["foo"]}}]}]' | jsonpipe
  /   []
  /0  {}
  /0/a        []
  /0/a/0      {}
  /0/a/0/b    {}
  /0/a/0/b/c  []
  /0/a/0/b/c/0        "foo"
In my own work, I've built up a suite of stream-based nested record processing tools that accept & produce JSON, protocol buffers, and a unix tab-delimited format. For the unix format it's been more useful to stick to the standard one-record-per-line thing, and let the user specify what fields to extract and their order.

Here's a depressing example of the fun you can have with new media and old unix tools, to give you some idea:

  $ mill io -r json -w texty -W fields=in_reply_to_screen_name

Re: Convert JSON to a Unix-friendly line-based format

#16

Post by bretthoerner »

jedsmith wrote: post #11
masklinn wrote: post #10

Earlier quoted context omitted.

> I'm also surprised that simplejson is used 2.6's json lacks simplejson's C accelerators [0] which makes it roughly 20 times slower than simplejson (2.7's simplejson is also ~50% slower than simplejson, as new performance optimizations were added in 2.1.0, as well as memoizations). Simplejson is also updated more often, which lead to it having less bugs, and interesting new features (the ability to natively serializ…

I elect to stick with the standard library in all cases, so I know how my software will perform everywhere without a redundant external dependency. Sticking with the standard library means you'll eventually get those improvements, too.

> a redundant external dependency

Is pip install that hard? Surely any deployed project is already managing a requirements.txt/buildout/setup.py/etc.

> means you'll eventually get those improvements

The C speedup extension for simplejson existed well before it was merged into Python stdlib as "json". Are you so sure you'll eventually get said improvement?

Re: Convert JSON to a Unix-friendly line-based format

#17

Post by masklinn »

jedsmith wrote: post #11
masklinn wrote: post #10

Earlier quoted context omitted.

> I'm also surprised that simplejson is used 2.6's json lacks simplejson's C accelerators [0] which makes it roughly 20 times slower than simplejson (2.7's simplejson is also ~50% slower than simplejson, as new performance optimizations were added in 2.1.0, as well as memoizations). Simplejson is also updated more often, which lead to it having less bugs, and interesting new features (the ability to natively serializ…

I elect to stick with the standard library in all cases, so I know how my software will perform everywhere without a redundant external dependency. Sticking with the standard library means you'll eventually get those improvements, too.

> Sticking with the standard library means you'll eventually get those improvements, too.

Not in 2.x, you won't.

Re: Convert JSON to a Unix-friendly line-based format

#18

Post by wnoise »

> Because the path components are separated by / characters, an object key like "abc/def" would result in ambiguous output. jsonpipe will throw an error if this occurs in your input, so that you can recognize and handle the issue. To mitigate the problem, you can choose a different path separator:

Ugh. I should never ever have to pick details of the format to work around content in the format. The only real solution is escaping, though it does add complexity.

Re: Convert JSON to a Unix-friendly line-based format

#19

Post by asymptotic »

wnoise wrote: post #18

> Because the path components are separated by / characters, an object key like "abc/def" would result in ambiguous output. jsonpipe will throw an error if this occurs in your input, so that you can recognize and handle the issue. To mitigate the problem, you can choose a different path separator: Ugh. I should never ever have to pick details of the format to work around content in the format. The only real solution…

Hahahah he uses a Unicode snowman as his example delimeter!

$ echo '{"abc/def": 123}' | jsonpipe -s '☃' ☃ {} ☃abc/def 123

I'm ready to face the day with a smile on my face.

Re: Convert JSON to a Unix-friendly line-based format

#20

Post by wladimir »

jedsmith wrote: post #11
masklinn wrote: post #10

Earlier quoted context omitted.

> I'm also surprised that simplejson is used 2.6's json lacks simplejson's C accelerators [0] which makes it roughly 20 times slower than simplejson (2.7's simplejson is also ~50% slower than simplejson, as new performance optimizations were added in 2.1.0, as well as memoizations). Simplejson is also updated more often, which lead to it having less bugs, and interesting new features (the ability to natively serializ…

I elect to stick with the standard library in all cases, so I know how my software will perform everywhere without a redundant external dependency. Sticking with the standard library means you'll eventually get those improvements, too.

Best of both worlds would be to try to import simplejson, and if that fails, fall back to the standard library json. As they have the same interface, that's trivial to do.
Post reply on HN