Text Processing in the Shell
blog.balthazar-rouberol.com
Text Processing in the Shell
1–10 of 109 posts
Re: Text Processing in the Shell
#2Just Don't. Unix-style text stream processing was super cool in 80s-90s but is born tech-debt today.
Re: Text Processing in the Shell
#3I really like that post, this is very good. I always loved NLP and that gave me a different angle. Where are you going now? I would love to know.
Re: Text Processing in the Shell
#4I wish man pages were this good!
Re: Text Processing in the Shell
#5Pleased add the "join"
Re: Text Processing in the Shell
#6A little suggestion for the authors, they mentioned xargs, I think [GNU parallel](https://www.gnu.org/software/parallel/) might work a mention too, since it is a kind of modern successor that can use many computers to run tasks.
Re: Text Processing in the Shell
#7Just Don't. Unix-style text stream processing was super cool in 80s-90s but is born tech-debt today.
what do you do instead?
Re: Text Processing in the Shell
#8Just Don't. Unix-style text stream processing was super cool in 80s-90s but is born tech-debt today.
what do you do instead?
Not OP, but I imagine he's bought into the Hadoop and other big data stories. Most data processing of large data sets can probably be done with standard UNIX tools.
There is nothing new under the sun, it's all just rebranded.
Re: Text Processing in the Shell
#9Re: Text Processing in the Shell
#10Earlier quoted context omitted.
what do you do instead?
If you do "object manipulation" e.g. columns/records/objects etc then at least Powershell offers some sanity.
Shameless plug but this is what I wrote murex for. Murex is a "UNIX" shell that's keeps enough similarities with POSIX that you can use it like a Bash REPL with (hopefully) minimal disruption but it breaks from POSIX compatibility where it makes sense. And one significant area it does break compatibility is how it handles structured data formats like JSON, S-Expressions, CSV's, and other tabulated data (to name a few).
It's designed from the ground up to support object manipulation while still retaining compatibility with the UNIX pipeline.
I do this by building a suite of builtin tools that are aware of structured data files (primarily because that information is passed down the pipeline as a data-type) but it still breaks into normal pipeline when forking an external executable.