Convert JSON to a Unix-friendly line-based format

#1

Post by adulau » Sun, Apr 10, 2011, 7:05 PM UTC

Convert JSON to a Unix-friendly line-based format
github.com

Re: Convert JSON to a Unix-friendly line-based format

#2

Post by haberman » Sun, Apr 10, 2011, 7:52 PM UTC

"Everyone I know prefers to work with JSON over XML, but sadly there is a sore lack of utilities of the quality or depth of html-xml-utils and XMLStarlet for actually processing JSON data in an automated fashion, short of writing an ad hoc processor in your favourite programming language."

Actually there is just such a suite of utilites! See https://github.com/benbernard/RecordStream

Re: Convert JSON to a Unix-friendly line-based format

#3

Post by tebeka » Sun, Apr 10, 2011, 8:01 PM UTC

cat foo.json | python -m json.tool

Re: Convert JSON to a Unix-friendly line-based format

#4

Post by brendano » Sun, Apr 10, 2011, 8:18 PM UTC

The use case is a bit different, but I wrote a little converter to TSV, adding in a header. I mostly use it for input into R, but I use it a lot. It only works for fairly flat JSON objects. https://github.com/brendano/tsvutils/blob/master/json2tsv

Re: Convert JSON to a Unix-friendly line-based format

#5

Post by adulau » Sun, Apr 10, 2011, 8:21 PM UTC

haberman wrote: post #2

"Everyone I know prefers to work with JSON over XML, but sadly there is a sore lack of utilities of the quality or depth of html-xml-utils and XMLStarlet for actually processing JSON data in an automated fashion, short of writing an ad hoc processor in your favourite programming language." Actually there is just such a suite of utilites! See https://github.com/benbernard/RecordStream

RecordStream seems more complete but requires much more dependencies. I really like the simplicity of jsonpipe and how easy is to use it:

      curl -s "http://feeds.delicious.com/v2/json/adulau  | jsonpipe -s "#"  | grep  "\#u" | cut -f2 | sed -e"s/\"//g" | xargs -d"\n" wget -r -l 1 –p –-convert-links

A simple example to take a local mirror of my last del.icio.us bookmarks...

Re: Convert JSON to a Unix-friendly line-based format

#6

Post by zpoley » Sun, Apr 10, 2011, 8:25 PM UTC

Here is my contribution to Unix friendly JSON command line tools (requires Node.js and NPM): https://github.com/zpoley/json-command. Here are a couple other good ones: http://kmkeen.com/jshon/ https://github.com/micha/jsawk

Re: Convert JSON to a Unix-friendly line-based format

#7

Post by vlisivka » Sun, Apr 10, 2011, 8:38 PM UTC

tebeka wrote: post #3

cat foo.json | python -m json.tool

  $ cat foo.json 
  { foo:"bar" }
  $ js -e "var doc=`cat foo.json`; print(doc.foo);"
  bar

Re: Convert JSON to a Unix-friendly line-based format

#8

Post by y0ghur7_xxx » Sun, Apr 10, 2011, 8:53 PM UTC

Why not just use Rhino or any other stand alone javascript interpreter?

http://www.mozilla.org/rhino/

a simple example

echo 'x=[1,2,3];x[1]'|js

Re: Convert JSON to a Unix-friendly line-based format

#9

Post by jedsmith » Sun, Apr 10, 2011, 9:01 PM UTC

    import os.path as p

Renaming imports to single letters bugs me. I went to see where it's used, and it isn't. Binding pyflakes to Cmd+S in TextMate was the best thing I ever did, and it would have caught this. pyflakes is seriously awesome in that role, and pylint before committing.

I'm also surprised that simplejson is used, instead of the built-in json in Python 2.6 and up. A good solution is:

    try: import json
    except ImportError: import simplejson as json

Re: Convert JSON to a Unix-friendly line-based format

#10

Post by masklinn » Sun, Apr 10, 2011, 9:09 PM UTC

jedsmith wrote: post #9

import os.path as p Renaming imports to single letters bugs me. I went to see where it's used, and it isn't. Binding pyflakes to Cmd+S in TextMate was the best thing I ever did, and it would have caught this. pyflakes is seriously awesome in that role, and pylint before committing. I'm also surprised that simplejson is used, instead of the built-in json in Python 2.6 and up. A good solution is: try: import json excep…

> I'm also surprised that simplejson is used

2.6's json lacks simplejson's C accelerators [0] which makes it roughly 20 times slower than simplejson (2.7's simplejson is also ~50% slower than simplejson, as new performance optimizations were added in 2.1.0, as well as memoizations). Simplejson is also updated more often, which lead to it having less bugs, and interesting new features (the ability to natively serialize decimals was added in 2.1 for instance)

simplejson should be the preferred import, with json as a fallback if available.

[0] 2.6's json is simplejson 1.9; 2.7's is 2.0.9; the latest release of simplejson is 2.1.3 and 2.1.4 is in preparation