What you need may be “pipeline +Unix commands” only

Post by IshKebab » Sun, Mar 03, 2019, 5:14 PM UTC

mirceal wrote: post #105

klhugo wrote: post #99

Earlier quoted context omitted.

I admire your work. Clever usage of unix tools is very handy. But for parsing text, do you really see that awk and Unix tools as a better solution then a simple python script? Although I admit that the key argument for Unix tools is that they don’t get updated. That sounds awful, but think about it, once it works, it works everywhere, no matters OS type, version or packages installed. That is something experienced pr…

parsing text is what a lot of these scripts/mini-pipelines do. the key argument for *nix tools is that they do one thing and only one thing extremely well. at a meta level these tools are units of functionality and you’re actually doing functional programming, on the command line, without realizing it.

Post by twic » Sun, Mar 03, 2019, 5:24 PM UTC

klhugo wrote: post #99

twic wrote: post #72

I've been a pipeline junkie for a long time, but i've only recently started to get into awk. The thing i can do with awk but not other tools is to write stateful filters, which accumulate information in associative arrays as they go. For example, if you want to do uniq without sorting the input, that's: awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }' This works best if the number of unique lines is small, either…

I admire your work. Clever usage of unix tools is very handy. But for parsing text, do you really see that awk and Unix tools as a better solution then a simple python script? Although I admit that the key argument for Unix tools is that they don’t get updated. That sounds awful, but think about it, once it works, it works everywhere, no matters OS type, version or packages installed. That is something experienced pr…

  import sys
  for line in sys.stdin:
    fields = line.split()
    # now you can do your logic

Post by mooreds » Sun, Mar 03, 2019, 5:29 PM UTC

lapnitnelav wrote: post #124

mooreds wrote: post #84

Earlier quoted context omitted.

> Turns out, most people don't want to have anything to do with a command prompt, even if the hard part has been done for you. That's been a pretty good lesson BTW. My experience has been that some folks are resistant to the command line (but I wouldn't say most). This is too bad because I feel like it's a crucial part of development. I even wrote a post about it: https://letterstoanewdeveloper.com/2019/02/04/learn-t…

In my specific case, it wasn't dev profiles but people on a spectrum going from I grok tech stuff to only browsing and the office suite. So obviously, whenever I started doing trainings on how to use the cli / the toolkit, I could see what is best described as mild panic. Which I get, CLI isn't inviting at all, it's pretty daunting, there's no real emphasis on safety (as in not breaking anything), it's far from being…

Post by jandrese » Sun, Mar 03, 2019, 5:32 PM UTC

jimmy_ruska wrote: post #38

* If it's simple transforms, use cli tools. * If it requires aggregation and it's small, use cli tools. * If this is data you're using over and over again then load it in the database and then do the cleaning, ELT. * If it's 2tb of data and under, still use bzip2, get splittable streams and pass it to gnu parallel. * If it requires massive aggregations or windows, use spark|flink|bleam. * If you need to repeatedly pr…

Post by forgotmypw7 » Sun, Mar 03, 2019, 5:47 PM UTC

rofo1 wrote: post #6

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…