Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Text Processing in the Shell

blog.balthazar-rouberol.com

Post reply on HN
Print view View original

91–100 of 109 posts

Re: Text Processing in the Shell

#91

Post by enriquto »

bryanrasmussen wrote: post #88
enriquto wrote: post #87

Earlier quoted context omitted.

oh my god, why?! just why!? What the world needs is the inverse program of "jc", where an unparseable json string is expanded into a flat list of lines all of the form "field.subfield=value"

ok well I can understand not wanting json when all you need is something simpler, but I'm not sure if I understand unparseable - I mean if it is JSON then it is parseable.

You can only parse json easily by using json libraries. Plain text, or "field=value" pairs, you can easily cut(1) or grep(1), or sed(1) to your pleasure. This is what I mean by parseable. Parseable trivially by tools that do not understand the format. I can also sed and awk json files, and I do, but it is extremely painful; and more often than not these files are nothing more than simple lists of variables with values, for which the use of json is a ridiculous overkill.

Re: Text Processing in the Shell

#92

Post by JdeBP »

dublin wrote: post #78
hnlmorg wrote: post #51

Earlier quoted context omitted.

> ... and pretty much all of those were addressed, [outside?] Unix, by the evolution of the 1960s terminal I/O model into the console I/O model during the 1980s. > Note that the lack of a layout system is only applicable to character-mode terminals. Block-mode terminals are a quite different kettle of fish. Indeed but the point isn't "are these solvable problems?" but rather "why are we still using archaic tech?" Des…

Check out Microsoft's new Windows terminal ( https://www.hanselman.com/blog/ItsTimeForYouToInstallWindows... ) It may be the most modern, capable, and yet compatible terminal I've come across, and works well regardless of the environment you want to run it in/with. (To be fair, a modern Windows terminal is a good 20 years overdue, but MS deserves credit for finally getting it right...)

It doesn't really support all of the basics as exemplified in https://github.com/jdebp/terminal-tests/blob/master/PowerShe... and https://github.com/jdebp/terminal-tests/blob/master/PowerShe... yet.

https://github.com/jdebp/terminal-tests/blob/master/PowerShe... is arguable, and depends from whether one sides with old actual DEC VTs or the ECMA-48:1986 standard, which the DEC VTs didn't keep up with. There are at least two existing terminal emulators that side with the new 1986 semantics, nowadays.

It really isn't anywhere near the most capable terminal emulator right now. This is acknowledged by its developers, and they even have long to-do lists of missing stuff, both as compared to a DEC VT and compared to the likes of XTerm.

Re: Text Processing in the Shell

#93

Post by bryanrasmussen »

enriquto wrote: post #91
bryanrasmussen wrote: post #88

Earlier quoted context omitted.

ok well I can understand not wanting json when all you need is something simpler, but I'm not sure if I understand unparseable - I mean if it is JSON then it is parseable.

You can only parse json easily by using json libraries. Plain text, or "field=value" pairs, you can easily cut(1) or grep(1), or sed(1) to your pleasure. This is what I mean by parseable. Parseable trivially by tools that do not understand the format. I can also sed and awk json files, and I do, but it is extremely painful; and more often than not these files are nothing more than simple lists of variables with value…

Ok, but I believe that your use of the word parseable diverges quite a bit from common usage.

Re: Text Processing in the Shell

#94

Post by msla »

JdeBP wrote: post #90
msla wrote: post #74

Earlier quoted context omitted.

> the evolution of the 1960s terminal I/O model into the console I/O model during the 1980s. Are you talking about MS-DOS-style memory-diddling to achieve things like colors and reverse video?

Read https://news.ycombinator.com/item?id=17238350 and https://news.ycombinator.com/item?id=22603878 in this very discussion.

So, to quote:

> Platforms with "consoles". These platforms provide a concept of a "console" to applications programs. Consoles support direct screen addressing, to the level of character cells at least, and are accessed through an API that is a first-class part of the overall system API.

> Platforms with "terminals". These platforms provide a concept of a "terminal" to applications programs. Terminals are not directly addressed, but are communicated with via byte stream communications protocols, involving control characters and control character sequences.

In short, you're deliberately being non-specific. I'd claim that a terminal system plus ncurses is a flexible and reasonably efficient console which is portable between many different systems, on the hosting end and on the client end. I could claim that the IBM block-mode "terminals" are consoles if the system is taken as a whole, but such things are markedly less flexible than what you can accomplish with ncurses, albeit more machine-efficient on the host side.

As far as first-class APIs go, you can argue with others about the kernel-vs-OS distinction, and the irrelevance of unbundling in the Open Source world. In short, shipping with ncurses is no more "odd" than shipping with Gtk or, say, a web browser.

Re: Text Processing in the Shell

#95

Post by Igrom »

enriquto wrote: post #87
BaltoRouberol wrote: post #29

Earlier quoted context omitted.

https://github.com/kellyjonbrazil/jc can come pretty handy there

oh my god, why?! just why!? What the world needs is the inverse program of "jc", where an unparseable json string is expanded into a flat list of lines all of the form "field.subfield=value"

If you search for "gron", you will find a family of such tools, e.g., https://github.com/tailhook/rust-gron.

Re: Text Processing in the Shell

#96

Post by enriquto »

Igrom wrote: post #95
enriquto wrote: post #87

Earlier quoted context omitted.

oh my god, why?! just why!? What the world needs is the inverse program of "jc", where an unparseable json string is expanded into a flat list of lines all of the form "field.subfield=value"

If you search for "gron", you will find a family of such tools, e.g., https://github.com/tailhook/rust-gron .

Thanks, that's just what I needed!

Re: Text Processing in the Shell

#97

Post by zmmmmm »

oweiler wrote: post #18
thekelvinliu wrote: post #7

Earlier quoted context omitted.

what do you do instead?

We use Groovy or Kotlin with KScript.

Groovy is particularly nice I find. Completely cross platform and supports command line args similar to Perl to enable inline / pipe style processing easily, but if you are using Java in your back end you can throw any of your business logic in there too and use those.

Re: Text Processing in the Shell

#98

Post by shp0ngle »

BaltoRouberol wrote: post #23
shp0ngle wrote: post #12

Sometimes I find it strange - in both good and bad way - that we are, in 2020, learning tools and languages designed and build in the 80s, with models and constraints of the time, with 40 years of layers of backwards compatibility, and actually going sometimes back to the 70s I am still learning tools designed around the constraints of teleprinters Sure, it’s the same on Windows side (and macOS side with their classi…

Author here. Quick sidenote: this is chapter 2 of a book aiming at teaching the terminal and related tooling (make, jq, regular expressions, etc) to beginners and people trying to become developers. I've been on the job for 10 years and I still use these tools daily. For example, I sent a PR yesterday that was adding a configuration entry on 150+ config files by using find, grep and sed. I'm not pretending these are…

Please don’t take it wrong, I like what I read so far! It was just my random observation

Re: Text Processing in the Shell

#99

Post by pixelmonkey »

shp0ngle wrote: post #12

Sometimes I find it strange - in both good and bad way - that we are, in 2020, learning tools and languages designed and build in the 80s, with models and constraints of the time, with 40 years of layers of backwards compatibility, and actually going sometimes back to the 70s I am still learning tools designed around the constraints of teleprinters Sure, it’s the same on Windows side (and macOS side with their classi…

If you think of UNIX tools the way a chef thinks about kitchen tools (like chefs’ knives and cutting boards, or mixing bowls and frying pans), you won’t be as surprised to be using stuff that dates back decades. Good design is built to last and to enhance the skills of an experienced craftsperson. I wrote a little on this here:

https://amontalenti.com/2012/06/22/unix-kitchen

Re: Text Processing in the Shell

#100

Post by hnlmorg »

dublin wrote: post #78
hnlmorg wrote: post #51

Earlier quoted context omitted.

> ... and pretty much all of those were addressed, [outside?] Unix, by the evolution of the 1960s terminal I/O model into the console I/O model during the 1980s. > Note that the lack of a layout system is only applicable to character-mode terminals. Block-mode terminals are a quite different kettle of fish. Indeed but the point isn't "are these solvable problems?" but rather "why are we still using archaic tech?" Des…

Check out Microsoft's new Windows terminal ( https://www.hanselman.com/blog/ItsTimeForYouToInstallWindows... ) It may be the most modern, capable, and yet compatible terminal I've come across, and works well regardless of the environment you want to run it in/with. (To be fair, a modern Windows terminal is a good 20 years overdue, but MS deserves credit for finally getting it right...)

If it's backwards compatible with TTYs then it probably hasn't solved any of the problems I mentioned and there's already a plethora of nice terminal emulators out there (which is why I say most of the "ugliness" is hidden from everyday users).

It should also be noted that Windows makes a few mistakes when it comes to terminal design too:

- For starters cmd.exe is both a shell and terminal emulator and it's impossible to separate the two.

- To compound things, many common commands like dir, rm, etc are shell builtins (this will be a throwback to DOS). So you cannot even use an alternative shell on Windows without having to either invoke cmd.exe or rewrite existing utilities.

- And if that wasn't bad enough, cmd.exe builtins do not read from STDIN. They instead use DOS syscalls to read keyboard input.

- As you've probably guessed, cmd.exe isn't the only culprit which does this. Any "Windows" command line program designed for or which uses DOS APIs will not follow the standard streams idiom. Any command line software written for NT, however will. So you end up having to write all sorts of really nasty hacks just to get the command line working on Windows (far far nastier than any of the hacks that happen on UNIX/Linux).

- Then you have Powershell, which is an entirely separate command line in its own right and largely - though not completely - incompatible with cmd.exe.

- And WSL, which is also incompatible with cmd.exe and Powershell.

At least on UNIX/Linux, you have one terminal methodology. From there you can use whichever terminal emulator you want, whichever shell you want, whichever programming language to write CLI tools and/or whichever CLI tools you want to download. Where as on Windows you have 4 competing standards which don't cooperate well.

Post reply on HN