Live data from Hacker News

Jo – a shell command to create JSON (2016)

jpmens.net

31–40 of 100 posts

Re: Jo – a shell command to create JSON (2016)

#31
Five minute job:

  $ ./jo foo=1 bar=2 obj=$(./jo -a 1 2 3 " def
  > ghi'
  {"foo":"abc\ndef\nghi"}

  $ cat jo
  #!/usr/local/bin/txr --lisp
  
  (define-option-struct jo-opts nil
    (a   array   :bool
         "Produce array instead of object")
    (nil help    :bool
         "Print this help"))
  
  (defvarl jo-name *load-path*)
  
  (defun json-val (str)
    (match-case str
      ("true" t)
      ("false" nil)
      ("null" 'null)
      (`{@nil` (get-json str))
      (`[@nil` (get-json str))
      (@else (iflet ((num (tofloat else)))
               num
               else))))
  
  (let ((o (new jo-opts)))
    o.(getopts *args*)
    (when o.help
      (put-line "Usage:\n")
      (put-line `  @{jo-name} [options] arg*`)
      o.(opthelp)
      (exit 0))
  
    (if o.array
      (let ((items [mapcar json-val o.out-args]))
        (put-jsonl (vec-list items)))
      (let ((pairs [mapcar (lambda (:match)
                             ((`@this=@that`) (list (json-val this) (json-val that)))
                             ((@else) (error "~a: arguments must be name=obj pairs" jo-name)))
                           o.out-args]))
        (put-jsonl ^#H(() ,*pairs)))))

Re: Jo – a shell command to create JSON (2016)

#32

I remember that when I worked at Google about a decade ago, there was this common saying: "If the first version of your shell script is more than five lines long, you should have written it in Python." I think there's a lot of truth in that. None of the examples presented in the article look better than had they been written in some existing scripting/programming language. In fact, had they been written in Python or…

For programs this trivial, startup time so dominates runtime, and Python's startup time is so incredibly awful, that you can often fork 10-20 low-overhead processes before Python even starts executing user code.

Re: Jo – a shell command to create JSON (2016)

#33

I remember that when I worked at Google about a decade ago, there was this common saying: "If the first version of your shell script is more than five lines long, you should have written it in Python." I think there's a lot of truth in that. None of the examples presented in the article look better than had they been written in some existing scripting/programming language. In fact, had they been written in Python or…

[deleted]

Re: Jo – a shell command to create JSON (2016)

#34
post #28
post #22

Earlier quoted context omitted.

then use -s : jo -- -s opaque_id=001979 {"opaque_id":"001979"}

jo opaque_id=$(command used in prod that returns digits + letters for three months and everything works just fine and then suddenly at 3am on a Sunday it returns 1979 and breaks everything) Avoid using this command in prod.

    jo -- -s a=123 -s b=00123 -s c="$(date)"
    {"a":"123","b":"00123","c":"Sat  5 Feb 21:29:08 GMT 2022"}
What's the issue?

Re: Jo – a shell command to create JSON (2016)

#35

I remember that when I worked at Google about a decade ago, there was this common saying: "If the first version of your shell script is more than five lines long, you should have written it in Python." I think there's a lot of truth in that. None of the examples presented in the article look better than had they been written in some existing scripting/programming language. In fact, had they been written in Python or…

> Even though Python isn't the fastest language out there, it's likely still faster than the shell command above.

Taking these two command lines:

   jo -p name=JP object=$(jo fruit=Orange point=$(jo x=10 y=20) number=17) sunday=false >/dev/null

   python -c 'import json;print(json.dumps({"name": "JP", "object": {"fruit": "Orange", "point": {"x": 10, "y": 20}, "number": 17}, "sunday": False}))' >/dev/null
For jo (x86_64, Rosetta2), python2 (x86_64, Rosetta2), jo (arm64), and python3 (arm64), running 1000 iterations, with `tai64n` doing the timing.

    2022-02-05 21:25:38.357228500 start-jo-x86
    2022-02-05 21:25:45.319337500 stop-jo
    2022-02-05 21:25:45.319338500 start-python2-x86
    2022-02-05 21:26:18.876235500 stop-python2-x86
    2022-02-05 21:26:18.876235500 start-jo-arm
    2022-02-05 21:26:22.316063500 stop-jo-arm
    2022-02-05 21:26:22.316064500 start-python3-arm
    2022-02-05 21:26:40.379063500 stop-python3-arm
I make it: 7s for jo-x86, 33.5s for python2-x86, 3.5s for jo-arm, 18s for python3-arm.

Test script is at https://pastebin.com/4tTVrDia

Re: Jo – a shell command to create JSON (2016)

#36

I remember that when I worked at Google about a decade ago, there was this common saying: "If the first version of your shell script is more than five lines long, you should have written it in Python." I think there's a lot of truth in that. None of the examples presented in the article look better than had they been written in some existing scripting/programming language. In fact, had they been written in Python or…

> Even though Python isn't the fastest language out there, it's likely still faster than the shell command above. Taking these two command lines: jo -p name=JP object=$(jo fruit=Orange point=$(jo x=10 y=20) number=17) sunday=false >/dev/null python -c 'import json;print(json.dumps({"name": "JP", "object": {"fruit": "Orange", "point": {"x": 10, "y": 20}, "number": 17}, "sunday": False}))' >/dev/null For jo (x86_64, Ro…

Now put a thousand of those JSON objects in a list, invoking jo for every element.

Re: Jo – a shell command to create JSON (2016)

#37

Earlier quoted context omitted.

> Even though Python isn't the fastest language out there, it's likely still faster than the shell command above. Taking these two command lines: jo -p name=JP object=$(jo fruit=Orange point=$(jo x=10 y=20) number=17) sunday=false >/dev/null python -c 'import json;print(json.dumps({"name": "JP", "object": {"fruit": "Orange", "point": {"x": 10, "y": 20}, "number": 17}, "sunday": False}))' >/dev/null For jo (x86_64, Ro…

Now put a thousand of those JSON objects in a list, invoking jo for every element.

That wasn't the claim made in the original post though, was it? The claim was that the Python snippet would be quicker than the jo snippet.

"Even though Python isn't the fastest language out there, it's likely still faster than the shell command above."

Which is most definitely is not - it's 5x slower.

(Probably not a huge issue in the real world if you're writing a shell script, mind, given that bash itself isn't a performance demon. But claims have to be tested.)

Re: Jo – a shell command to create JSON (2016)

#38
post #19

>Bam! Jo tries to be clever about types and knows null, booleans, strings and numbers. I'm very skeptical of this. If I put x=001979 in as a value I dont think I want you trying to guess if that's supposed to be an integer or a string. This sounds like the Norway Problem waiting to happen.

This kind of thing has its place and can be useful, but I think there should be options to enable/disable such magic. Personally I'd lean towards it being opt-in, but I think with the cli it's a lot harder to not have it opt-out. Everythint is a string in the cli, but not so much when it comes to json. That's why I think it makes sense, providing you can disable the magic.

Re: Jo – a shell command to create JSON (2016)

#39
post #34
post #28

Earlier quoted context omitted.

jo opaque_id=$(command used in prod that returns digits + letters for three months and everything works just fine and then suddenly at 3am on a Sunday it returns 1979 and breaks everything) Avoid using this command in prod.

jo -- -s a=123 -s b=00123 -s c="$(date)" {"a":"123","b":"00123","c":"Sat 5 Feb 21:29:08 GMT 2022"} What's the issue?

The problem is avoiding your surprise ruined Sunday night three months from now is predicated on you not forgetting to put the -s in.

Implicit typing sucks: https://www.destroyallsoftware.com/talks/wat

Re: Jo – a shell command to create JSON (2016)

#40

Earlier quoted context omitted.

Now put a thousand of those JSON objects in a list, invoking jo for every element.

That wasn't the claim made in the original post though, was it? The claim was that the Python snippet would be quicker than the jo snippet. "Even though Python isn't the fastest language out there, it's likely still faster than the shell command above." Which is most definitely is not - it's 5x slower. (Probably not a huge issue in the real world if you're writing a shell script, mind, given that bash itself isn't a…

That’s because you’re making a false assumption about the environment prior to executing the statement.

If you are in a shell session and have to choose between executing python -c or calling jo, the latter is faster as you’ve demonstrated. But that’s not a realistic assumption.

Statements like these are almost certainly part of some combined work. The data you’re feeding to jo comes from somewhere. Its output is written somewhere.

You can’t convince me that if you’re already inside some Python script, that invoking json.dumps() is slower than calling jo from within a shell script.

At no point did I claim that launching Python AND running that json.dumps() is faster than running that shell command. I only stated that the json.dumps() is.

Post reply on HN