Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Text Processing in the Shell

blog.balthazar-rouberol.com

Post reply on HN
Print view View original

71–80 of 109 posts

Re: Text Processing in the Shell

#71

Post by fnord123 »

bmn__ wrote: post #55
fnord123 wrote: post #22

This is cool for English text. But once you get Unicode with various ways to represent é, whew lad. This get shitty quickly in the shell.

English is Unicode. Pretending otherwise would be quite naïve. https://www.azabani.com/pages/gbu/#slide4

You claim is not clear.

"Unicode with various ways to represent é" is a shit show to parse using shell tools. e.g. Try scraping Spanish language Twitter feeds. When I have done this kind of work, I made a tool to canonicalize glyphs and had to put it between every step of a pipeline.

Re: Text Processing in the Shell

#72

Post by overgard »

clircle wrote: post #60

I'll preface my question by saying that I'm not a dev. Why do this kind of work in the shell? Isn't it better to do this in a programming language that can run on all operating systems? What are Windows users supposed to do?

As a dev: I don't know. It's a well written article and this stuff can be handy in a pinch, but I've yet to see many real world scenarios where a complicated shell script is a good idea. Most of these examples, I would probably rather write a five line python script to ingest the data into sqlite and then use actual queries.

Re: Text Processing in the Shell

#73

Post by robenkleene »

shp0ngle wrote: post #12

Sometimes I find it strange - in both good and bad way - that we are, in 2020, learning tools and languages designed and build in the 80s, with models and constraints of the time, with 40 years of layers of backwards compatibility, and actually going sometimes back to the 70s I am still learning tools designed around the constraints of teleprinters Sure, it’s the same on Windows side (and macOS side with their classi…

Why do you find this strange? There’s a pattern across all fields: Different paradigms compete early on, and the best paradigm wins and we build on it incrementally for years (forever?) after. Every once in awhile a new workflow emerges based on increases in hardware capabilities: Ableton Live, Adobe Lightroom, and Figma are all examples of this. Notably these new tools usually split the market rather than replace the previous version: Logic, Photoshop, and Sketch are still popular.

Re: Text Processing in the Shell

#74

Post by msla »

JdeBP wrote: post #46
hnlmorg wrote: post #34

Earlier quoted context omitted.

In fairness to the OP, there's a lot of cruft in the way terminal work that really isn't necessary any more but need to be there because of backwards compatibility. Such as - formatting being in-lined via ANSI escape sequences, - and there's a massive disparity between what escape sequences terminal emulators support, - control codes being part of the same character set as printable characters, - changing the behavio…

... and pretty much all of those were addressed, outwith Unix, by the evolution of the 1960s terminal I/O model into the console I/O model during the 1980s. You even forgot to mention one of the things that was addressed: input. Terminal I/O input, done properly, requires a full ECMA-48 decoder state machine, with bodges to accommodate non-conformant warts from the Linux KVT, SCO Console, and RXVT. This is all too of…

> the evolution of the 1960s terminal I/O model into the console I/O model during the 1980s.

Are you talking about MS-DOS-style memory-diddling to achieve things like colors and reverse video?

Re: Text Processing in the Shell

#75

Post by nailer »

gpanders wrote: post #65
nailer wrote: post #49

Text processing is scraping. Modern shells have structured output from their stdlib, allow you to pipe to 'where' and 'select', and can read JSON, yaml, etc natively.

Which shells in particular are you referring to?

Mainly pwsh and nushell

Re: Text Processing in the Shell

#76

Post by nostoc »

overgard wrote: post #72
clircle wrote: post #60

I'll preface my question by saying that I'm not a dev. Why do this kind of work in the shell? Isn't it better to do this in a programming language that can run on all operating systems? What are Windows users supposed to do?

As a dev: I don't know. It's a well written article and this stuff can be handy in a pinch, but I've yet to see many real world scenarios where a complicated shell script is a good idea. Most of these examples, I would probably rather write a five line python script to ingest the data into sqlite and then use actual queries.

The use case is to quickly handle ad hoc scenarios.

If you need to quickly extract something from a csv, you could break out python, or import it into a database, but using cut and grep (or csv-tools) will take 5 seconds.

The point is if you need to do a specific task many times, do it in a programming language. But if you have an ad hoc task, you're saving a lot of time by being proficient in the shell.

Re: Text Processing in the Shell

#77

Post by 411111111111111 »

i was just missing a small note akin to "which becomes all the more powerful by combining these commands" and then make a totally readable example such as this one

     some_file=example.sh; tail -n +$(( $(wc -l $some_file | grep -o "[0-9]\+") - 5 )) $some_file

Re: Text Processing in the Shell

#78

Post by dublin »

hnlmorg wrote: post #51
JdeBP wrote: post #46

Earlier quoted context omitted.

... and pretty much all of those were addressed, outwith Unix, by the evolution of the 1960s terminal I/O model into the console I/O model during the 1980s. You even forgot to mention one of the things that was addressed: input. Terminal I/O input, done properly, requires a full ECMA-48 decoder state machine, with bodges to accommodate non-conformant warts from the Linux KVT, SCO Console, and RXVT. This is all too of…

> ... and pretty much all of those were addressed, [outside?] Unix, by the evolution of the 1960s terminal I/O model into the console I/O model during the 1980s. > Note that the lack of a layout system is only applicable to character-mode terminals. Block-mode terminals are a quite different kettle of fish. Indeed but the point isn't "are these solvable problems?" but rather "why are we still using archaic tech?" Des…

Check out Microsoft's new Windows terminal (https://www.hanselman.com/blog/ItsTimeForYouToInstallWindows... ) It may be the most modern, capable, and yet compatible terminal I've come across, and works well regardless of the environment you want to run it in/with. (To be fair, a modern Windows terminal is a good 20 years overdue, but MS deserves credit for finally getting it right...)

Re: Text Processing in the Shell

#79

Post by dublin »

a3n wrote: post #28
shp0ngle wrote: post #12

Sometimes I find it strange - in both good and bad way - that we are, in 2020, learning tools and languages designed and build in the 80s, with models and constraints of the time, with 40 years of layers of backwards compatibility, and actually going sometimes back to the 70s I am still learning tools designed around the constraints of teleprinters Sure, it’s the same on Windows side (and macOS side with their classi…

> and actually going sometimes back to the 70s Try 1725, especially for those of us who (still) set their editors to favor 80 columns. https://en.m.wikipedia.org/wiki/Punched_card

Remember to start your FORTRAN statements in Column 7!

Re: Text Processing in the Shell

#80

Post by dublin »

Koshkin wrote: post #57
ekianjo wrote: post #38

Earlier quoted context omitted.

> I am still learning tools designed around the constraints of teleprinters We are still reading books using a 2k year old alphabet to represent ideas. Not sure why it would be surprising that text manipulation is still the norm pretty much in everything we do, including computing.

On the other hand, I assume you do not ride on a horseback to where you work.

No, but that would be my dream job...
Post reply on HN