Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Text Processing in the Shell

blog.balthazar-rouberol.com

Post reply on HN
Print view View original

51–60 of 109 posts

Re: Text Processing in the Shell

#51

Post by hnlmorg »

JdeBP wrote: post #46
hnlmorg wrote: post #34

Earlier quoted context omitted.

In fairness to the OP, there's a lot of cruft in the way terminal work that really isn't necessary any more but need to be there because of backwards compatibility. Such as - formatting being in-lined via ANSI escape sequences, - and there's a massive disparity between what escape sequences terminal emulators support, - control codes being part of the same character set as printable characters, - changing the behavio…

... and pretty much all of those were addressed, outwith Unix, by the evolution of the 1960s terminal I/O model into the console I/O model during the 1980s. You even forgot to mention one of the things that was addressed: input. Terminal I/O input, done properly, requires a full ECMA-48 decoder state machine, with bodges to accommodate non-conformant warts from the Linux KVT, SCO Console, and RXVT. This is all too of…

> ... and pretty much all of those were addressed, [outside?] Unix, by the evolution of the 1960s terminal I/O model into the console I/O model during the 1980s.

> Note that the lack of a layout system is only applicable to character-mode terminals. Block-mode terminals are a quite different kettle of fish.

Indeed but the point isn't "are these solvable problems?" but rather "why are we still using archaic tech?"

Designing a solution to those problems is actually the easy part. It is shifting the ecosystem away from TTYs that's hard.

> You even forgot to mention

It wasn't intended as an exhaustive list :) There's plenty more issues I hadn't raised.

---

In an ideal world I'd love to see UNIX terminals reinvented. The reality is things are "good enough" for most people that they simply don't notice most of the issues and migrating to the next evolution of UNIX terminals would mean a break in backwards compatibility which will be more disruptive (initially in a negative way) than making do with the warts we currently have.

Re: Text Processing in the Shell

#52

Post by thesuperbigfrog »

latenightcoding wrote: post #47

OP you should mention perl one-liners in upcoming chapters https://catonmat.net/introduction-to-perl-one-liners

Agreed. Perl makes it easy to do complex text processing and can replace many individual command line text processing tools.

Recent versions of Perl also support UTF-8 so they can support text processing in different natural languages or internationalization needs. See https://en.wikibooks.org/wiki/Perl_Programming/Unicode_UTF-8

Re: Text Processing in the Shell

#53

Post by matvore »

seemslegit wrote: post #37
thekelvinliu wrote: post #7

Earlier quoted context omitted.

what do you do instead?

Use any modern programming language.

Modern programming language features are overrated. I prefer to choose the tools that are the most portable and involve the least social friction (i.e. something that is widely understood by people I work with).

For "fun" projects (stuff I'm not paid for) and workflow optimization, I just care about portability, which means C (with heavy use of the C stream library, a simple collections library of about 200 LOC, and occasionally POSIX syscalls) and shell scripting. After spending a lot of time learning languages as a hobby, I just don't believe the dark corners and warts of C and shell scripting are any worse than other languages.

Re: Text Processing in the Shell

#54

Post by bmn__ »

thesuperbigfrog wrote: post #52
latenightcoding wrote: post #47

OP you should mention perl one-liners in upcoming chapters https://catonmat.net/introduction-to-perl-one-liners

Agreed. Perl makes it easy to do complex text processing and can replace many individual command line text processing tools. Recent versions of Perl also support UTF-8 so they can support text processing in different natural languages or internationalization needs. See https://en.wikibooks.org/wiki/Perl_Programming/Unicode_UTF-8

> Recent versions of Perl

Make sure it's recent enough and released after 2002!

Re: Text Processing in the Shell

#56

Post by thesuperbigfrog »

bmn__ wrote: post #54
thesuperbigfrog wrote: post #52

Earlier quoted context omitted.

Agreed. Perl makes it easy to do complex text processing and can replace many individual command line text processing tools. Recent versions of Perl also support UTF-8 so they can support text processing in different natural languages or internationalization needs. See https://en.wikibooks.org/wiki/Perl_Programming/Unicode_UTF-8

> Recent versions of Perl Make sure it's recent enough and released after 2002!

I was thinking of Perl versions 5.14 (released in 2011) and later since that release fixed several Unicode-related bugs.

There have been other improvements and fixes in the versions up to 5.30, so the Unicode support now is pretty transparent.

Many of the classic command line text processing tools are not Unicode aware.

Re: Text Processing in the Shell

#57

Post by Koshkin »

ekianjo wrote: post #38
shp0ngle wrote: post #12

Sometimes I find it strange - in both good and bad way - that we are, in 2020, learning tools and languages designed and build in the 80s, with models and constraints of the time, with 40 years of layers of backwards compatibility, and actually going sometimes back to the 70s I am still learning tools designed around the constraints of teleprinters Sure, it’s the same on Windows side (and macOS side with their classi…

> I am still learning tools designed around the constraints of teleprinters We are still reading books using a 2k year old alphabet to represent ideas. Not sure why it would be surprising that text manipulation is still the norm pretty much in everything we do, including computing.

On the other hand, I assume you do not ride on a horseback to where you work.

Re: Text Processing in the Shell

#58

Post by ken »

enriquto wrote: post #21
shp0ngle wrote: post #12

Sometimes I find it strange - in both good and bad way - that we are, in 2020, learning tools and languages designed and build in the 80s, with models and constraints of the time, with 40 years of layers of backwards compatibility, and actually going sometimes back to the 70s I am still learning tools designed around the constraints of teleprinters Sure, it’s the same on Windows side (and macOS side with their classi…

The English language is several hundreds of years old, and the ADN is several hundreds of millions years old. Everything we are based on is legacy; I do not see why it should be different in computing.

And English is terrible as a language! We’ve been making fun of how bad it is for centuries.

Compare languages like Turkish or Korean, which are still natural languages but got a well-thought-out tune-up in the not-so-distant past.

Why can’t my language have a pluralization rule (for example) that’s so simple and regular it takes 1 minute to teach, and 3 minutes to master? Or an alphabet that looks like how it is pronounced, so we don’t have to waste hours each week as children memorizing thousands of special cases? This is absurd.

Re: Text Processing in the Shell

#60

Post by clircle »

I'll preface my question by saying that I'm not a dev.

Why do this kind of work in the shell? Isn't it better to do this in a programming language that can run on all operating systems? What are Windows users supposed to do?

Post reply on HN