Text Processing in the Shell • Hacker News Board

Re: Text Processing in the Shell

#21

Post by enriquto » Tue, Mar 17, 2020, 9:13 AM UTC

shp0ngle wrote: post #12

Sometimes I find it strange - in both good and bad way - that we are, in 2020, learning tools and languages designed and build in the 80s, with models and constraints of the time, with 40 years of layers of backwards compatibility, and actually going sometimes back to the 70s I am still learning tools designed around the constraints of teleprinters Sure, it’s the same on Windows side (and macOS side with their classi…

The English language is several hundreds of years old, and the ADN is several hundreds of millions years old. Everything we are based on is legacy; I do not see why it should be different in computing.

Re: Text Processing in the Shell

#22

Post by fnord123 » Tue, Mar 17, 2020, 9:15 AM UTC

This is cool for English text. But once you get Unicode with various ways to represent é, whew lad. This get shitty quickly in the shell.

Re: Text Processing in the Shell

#23

Post by BaltoRouberol » Tue, Mar 17, 2020, 9:21 AM UTC

shp0ngle wrote: post #12

Sometimes I find it strange - in both good and bad way - that we are, in 2020, learning tools and languages designed and build in the 80s, with models and constraints of the time, with 40 years of layers of backwards compatibility, and actually going sometimes back to the 70s I am still learning tools designed around the constraints of teleprinters Sure, it’s the same on Windows side (and macOS side with their classi…

Author here. Quick sidenote: this is chapter 2 of a book aiming at teaching the terminal and related tooling (make, jq, regular expressions, etc) to beginners and people trying to become developers.

I've been on the job for 10 years and I still use these tools daily. For example, I sent a PR yesterday that was adding a configuration entry on 150+ config files by using find, grep and sed.

I'm not pretending these are the only tools that exist, but darn are they handy sometimes, and good to have in your toolbelt.

Re: Text Processing in the Shell

#24

Post by kuschku » Tue, Mar 17, 2020, 9:25 AM UTC

thekelvinliu wrote: post #7

seemslegit wrote: post #2

Just Don't. Unix-style text stream processing was super cool in 80s-90s but is born tech-debt today.

what do you do instead?

Output data as JSON, manipulate and show it with jq

Re: Text Processing in the Shell

#25

Post by [deleted] » Tue, Mar 17, 2020, 9:34 AM UTC

[deleted]

Re: Text Processing in the Shell

#26

Post by zokier » Tue, Mar 17, 2020, 9:43 AM UTC

fnord123 wrote: post #22

This is cool for English text. But once you get Unicode with various ways to represent é, whew lad. This get shitty quickly in the shell.

I would generalize that further and say that shell processing works well when your data is well-behaved wrt the thing you are trying to do. But once you start venturing into untrusted data, escaping, and yes, Unicode, things can become hairy. So basically you should always be doing some sanity checks when working in shell

Re: Text Processing in the Shell

#27

Post by bryanrasmussen » Tue, Mar 17, 2020, 9:43 AM UTC

kuschku wrote: post #24

thekelvinliu wrote: post #7

Earlier quoted context omitted.

what do you do instead?

Output data as JSON, manipulate and show it with jq

not all data comes from sources that you control and have chosen how to output.

Re: Text Processing in the Shell

#28

Post by a3n » Tue, Mar 17, 2020, 9:46 AM UTC

shp0ngle wrote: post #12

Sometimes I find it strange - in both good and bad way - that we are, in 2020, learning tools and languages designed and build in the 80s, with models and constraints of the time, with 40 years of layers of backwards compatibility, and actually going sometimes back to the 70s I am still learning tools designed around the constraints of teleprinters Sure, it’s the same on Windows side (and macOS side with their classi…

> and actually going sometimes back to the 70s

Try 1725, especially for those of us who (still) set their editors to favor 80 columns.

https://en.m.wikipedia.org/wiki/Punched_card

Re: Text Processing in the Shell

#29

Post by BaltoRouberol » Tue, Mar 17, 2020, 9:56 AM UTC

bryanrasmussen wrote: post #27

kuschku wrote: post #24

Earlier quoted context omitted.

Output data as JSON, manipulate and show it with jq

not all data comes from sources that you control and have chosen how to output.

https://github.com/kellyjonbrazil/jc can come pretty handy there

Re: Text Processing in the Shell

#30

Post by RMPR » Tue, Mar 17, 2020, 10:03 AM UTC

pletnes wrote: post #13

RMPR wrote: post #6

A little suggestion for the authors, they mentioned xargs, I think [GNU parallel]( https://www.gnu.org/software/parallel/ ) might work a mention too, since it is a kind of modern successor that can use many computers to run tasks.

This you have to install, xargs is everywhere. Also with the -P flag you can parallelize the most common cases.

I agree on the install part, but xargs -P still run on the same host and I was addressing the multiple hosts feature of parallel.