Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Text Processing in the Shell

blog.balthazar-rouberol.com

Post reply on HN
Print view View original

31–40 of 109 posts

Re: Text Processing in the Shell

#31

Post by zokier »

throwaway_pdp09 wrote: post #15
shp0ngle wrote: post #12

Sometimes I find it strange - in both good and bad way - that we are, in 2020, learning tools and languages designed and build in the 80s, with models and constraints of the time, with 40 years of layers of backwards compatibility, and actually going sometimes back to the 70s I am still learning tools designed around the constraints of teleprinters Sure, it’s the same on Windows side (and macOS side with their classi…

I find these posts annoying that say we're anchored in past tech and we must be able to do better, but never suggest how. Please make a suggestion or two. (NB ultimately our character handling is based on 'writing' which goes back thousands of years, not 50, and it survives well).

Structured data and tools that operate on that structure. Maybe steal a thing or two from LISP machines. I remember the one time I had to work on an AS/400 (iirc) how interesting it was that one of the fundamental building blocks were records instead of dumb files. PowerShell has interesting ideas too, but it'd need to be far more built-in and integrated to the OS to truly shine.

Re: Text Processing in the Shell

#32

Post by zokier »

Lio wrote: post #17
shp0ngle wrote: post #12

Sometimes I find it strange - in both good and bad way - that we are, in 2020, learning tools and languages designed and build in the 80s, with models and constraints of the time, with 40 years of layers of backwards compatibility, and actually going sometimes back to the 70s I am still learning tools designed around the constraints of teleprinters Sure, it’s the same on Windows side (and macOS side with their classi…

It’s because fundamentally, when you get right down to it, we’re manipulating text files. Text is a very dense way to represent logic and ideas. This is not like legacy software that you can rewrite. You can add a GUI based on ideas from the late 70s/80s if you like but you’re still unlikely to come up with a more succinct way to represent logic than can be held in a text file. So it follows that small tools that dea…

I'd say that fundamentally we are manipulating streams or arrays of numbers. Everything is built on top of that. Each character in a tex is a number. Each pixel is a number. Each sample of sound is a number. For computers how the number ends up being transferred to the puny meatbag is completely incidental.

In more practical terms at least for me, vast majority of stuff I mangle through shell is not really text but structured data, typically either some sort of tree or a table, or something in-between. Sometimes the structure is more ad-hoc, sometimes it is very rigidly defined, but it's still there

Re: Text Processing in the Shell

#33

Post by enriquto »

kuschku wrote: post #24
thekelvinliu wrote: post #7

Earlier quoted context omitted.

what do you do instead?

Output data as JSON, manipulate and show it with jq

I hate jq with the power of a million suns. The whole point of text through pipelines is that data can be processed by tools that do not understand it. Formatting it in xml or json (there's no difference) breaks this beautiful orthogonality, and forces all the intermediate tools to deal with whatever the stupid markup du jour happens to be.

Re: Text Processing in the Shell

#34

Post by hnlmorg »

enriquto wrote: post #21
shp0ngle wrote: post #12

Sometimes I find it strange - in both good and bad way - that we are, in 2020, learning tools and languages designed and build in the 80s, with models and constraints of the time, with 40 years of layers of backwards compatibility, and actually going sometimes back to the 70s I am still learning tools designed around the constraints of teleprinters Sure, it’s the same on Windows side (and macOS side with their classi…

The English language is several hundreds of years old, and the ADN is several hundreds of millions years old. Everything we are based on is legacy; I do not see why it should be different in computing.

In fairness to the OP, there's a lot of cruft in the way terminal work that really isn't necessary any more but need to be there because of backwards compatibility. Such as

- formatting being in-lined via ANSI escape sequences,

- and there's a massive disparity between what escape sequences terminal emulators support,

- control codes being part of the same character set as printable characters,

- changing the behaviour of the TTY requires either terminal emulator support or OS support depending on the behaviour you require because the TTYs are defined partially via kernel drivers (which requires syscalls to alter) and partially by escape sequences,

- and in the case of kernel behaviour, those syscalls vary from one OS to another. Some OS's don't even support from TTY behaviours that other OSs do so you can't even guarantee that logic is cross platform and that you just wrap around specific differences in syscalls,

- resizing terminals UIs can be a nightmare -- often requiring capturing RPC signals and redrawing -- because there's no native layout system for drawing to the TTY,

This isn't meant as a criticism though because there's a lot the general design of terminals gets right (eg the kernel driver for TTY allows us to kill processes over remote shells like SSH and mosh). But I think terminals are one of those things that work "good enough" that most of the ugliness is hidden from everyday users. However if we were to redesign UNIX terminals from the ground up there is a lot of things most engineers would like to change and of lot of places where things could be improved. Like having an out-of-band channel for sending meta-data describing the pipeline but shouldn't be mixed in with the byte stream.

Re: Text Processing in the Shell

#38

Post by ekianjo »

shp0ngle wrote: post #12

Sometimes I find it strange - in both good and bad way - that we are, in 2020, learning tools and languages designed and build in the 80s, with models and constraints of the time, with 40 years of layers of backwards compatibility, and actually going sometimes back to the 70s I am still learning tools designed around the constraints of teleprinters Sure, it’s the same on Windows side (and macOS side with their classi…

> I am still learning tools designed around the constraints of teleprinters

We are still reading books using a 2k year old alphabet to represent ideas. Not sure why it would be surprising that text manipulation is still the norm pretty much in everything we do, including computing.

Re: Text Processing in the Shell

#39

Post by spurdoman77 »

seemslegit wrote: post #37
thekelvinliu wrote: post #7

Earlier quoted context omitted.

what do you do instead?

Use any modern programming language.

The gnu tools are super fast and do the job. Ive seen people spinning their own solutions which end up being super slow and arguable take more time to develop.

With any gnu tools it is great that they will stay there for your life and are usually by default installed on every system

Re: Text Processing in the Shell

#40

Post by dejj »

seemslegit wrote: post #2

Just Don't. Unix-style text stream processing was super cool in 80s-90s but is born tech-debt today.

I love to cook 5-course meals for friends and family. But sometimes when I'm all alone by myself, I slice up a bun and slap meat on it. It's as fast and tastes just as good as in the 80ies.
Post reply on HN