Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Become Shell Literate

drewdevault.com

Post reply on HN
Print view View original

301–310 of 341 posts

Re: Become Shell Literate

#301

Post by enriquto »

em500 wrote: post #289
enriquto wrote: post #152

Earlier quoted context omitted.

> Read a CSV file, foo.csv, and sort by the sum of the 4th and 7th columns (which are integers): Yes, that's a much better example. Here the "obvious" shell solution involves using awk to compute the sum of the two columns and putting it as the first field, sorting by the first field, and then removing that field. I guess a "rosetta stone" of such examples (e.g. on the frontpage of the site) would make a strong case…

Concretely, awk -F, -v OFS=, {print $2+$3, $1, $2, $3}' foo.csv | sort -n | cut -d, -f2-

I think this code does not fulfill the task: you need to sum columns 4 and 7, and keep the rest of the data intact in the output.

Re: Become Shell Literate

#302

Post by Nextgrid »

ddevault wrote: post #248
hiq wrote: post #202

Nice post, but 2020 and still using an unauthenticated connection (plain netcat) to transfer data including executables (if I understood correctly) is not something you can justify when there are plenty of safe alternatives (e.g. wormhole, syncthing). Maybe the recipient and the sender just compare hashes via a secure channel, but there's no mention of this.

The files were transferred via yggdrasil. In any case, it would have to be an unusually clever attacker to identify executables in a tarball being transmitted unwrapped via TCP on a port chosen out of a hat, and intercept it to do something malicious.

Malice is not the only concern. There was a post on here a couple days ago where AT&T was corrupting certain packets, and due to either luck of the TCP checksums being recalculated on the faulty equipment itself the bad packets were actually considered valid by the recipient.

Re: Become Shell Literate

#303

Post by eeskildsen »

filmgirlcw wrote: post #270
cjohnson318 wrote: post #257

Earlier quoted context omitted.

If you work with mechanical or electrical engineers, then you're going to run into Windows machines pretty quick, and then you need to convince them to install minGW to run your shell scripts. Sure, shell scripts are portable between Mac and *nix, but you're leaving out a large elephant.

Hopefully the next rise or WSL, Windows Terminal, and the OSS PowerShell will help that. I’ve anecdotally noticed more people using or willing to use PowerShell in some scenarios, now that it’s at parity on Mac and Linux (there are a few things in old-school classic Windows PowerShell that don’t work on Mac/Linux but the future direction and support is all x-platform) and it’s certainly easier to get a Linux shell ru…

Cross-platform PowerShell is fantastic. It's one of the first things I install on a new Linux box nowadays. I don't think I could go back to a string-based shell. Objects are so much cleaner and easier to manipulate.

Plus you get a huge standard library out of the box with its .NET integration.

It's really impressive what Microsoft has done with it.

Re: Become Shell Literate

#304

Post by hiq »

ddevault wrote: post #248
hiq wrote: post #202

Nice post, but 2020 and still using an unauthenticated connection (plain netcat) to transfer data including executables (if I understood correctly) is not something you can justify when there are plenty of safe alternatives (e.g. wormhole, syncthing). Maybe the recipient and the sender just compare hashes via a secure channel, but there's no mention of this.

The files were transferred via yggdrasil. In any case, it would have to be an unusually clever attacker to identify executables in a tarball being transmitted unwrapped via TCP on a port chosen out of a hat, and intercept it to do something malicious.

> The files were transferred via yggdrasil.

I wish you would have at least mentioned it in your post that you were indeed relying on the security of the network and that nc by itself is not secure.

I'm not saying it would have been a real risk in your particular case, but giving these examples on a public website is basically saying "see, you can do it like this". I'm not saying you don't know better, but some of your readers might not, and may replicate your unsafe (without the underlying assumptions) code where the risk is higher.

This is especially bad given that there are safe alternatives with equal or better ergonomics.

Re: Become Shell Literate

#305

Post by rakoo »

sime2009 wrote: post #296
rakoo wrote: post #194

It still strikes me that after all this time, the next step in the direction of the author is to use an editor like ACME ( https://en.wikipedia.org/wiki/Acme_(text_editor) ): use the shell to its maximum power, where the output of a command can be edited and used to control the existing interface. Take for instance this video: https://www.youtube.com/watch?v=4djoOiLste0 The author shows how the output of "git status"…

Your git example is very similar to what can be done in my terminal, [Extraterm]( https://extraterm.org/ ). See demo here: https://extraterm.org/features.html#reusing-command-output Thanks for the tip about Acme. I'm always looking for ideas about how to improve integration between the terminal based work flows and other applications, command line, GUI, or others.

Wow, I've been looking for a terminal that does this sort of thing for a while, nice job! And it's beautiful !

Re: Become Shell Literate

#306

Post by sqlhelpthrow »

jahbrewski wrote: post #251
simonw wrote: post #89

Learning to use the shell and learning SQL are the two skills that have stayed relevant throughout my entire 20 year career. Other tools and languages come and go, but the shell and SQL just keep on providing value.

Any resources/recommendations for truly learning SQL? I feel pretty comfortable with the basics, but could certainly stand to improve my SQL toolkit!

Haven't tried it, but have heard high praises here in HN for Jennifer Widom's SQL course. It's now on edX.

https://cs.stanford.edu/people/widom/

https://www.edx.org/bio/jennifer-widom

Re: Become Shell Literate

#307

Post by em500 »

enriquto wrote: post #301
em500 wrote: post #289

Earlier quoted context omitted.

Concretely, awk -F, -v OFS=, {print $2+$3, $1, $2, $3}' foo.csv | sort -n | cut -d, -f2-

I think this code does not fulfill the task: you need to sum columns 4 and 7, and keep the rest of the data intact in the output.

You're right, that should be

      awk -F, -v OFS=, '{print $4+$7, $0}' foo.csv  |  sort -n -k1 -t, |  cut -f2- -d,
Explanation:

awk -F, -v OFS=, sets the input and output column separator to comma, '{print $4+$7, $0}' outputs the sum of column 4 and 7 before the rest of the line.

sort -n -k1 -t, sorts the file numerically on column 1, with comma separator.

cut -f2- -d, removes column 1, with comma separator.

This is of course not robust for general CSV files, but I don't think OPs marcel is either. A robust solution requires a proper CSV parser.

The biggest warts I see in the classic unix solution is that all the tools use different flags for the field separator.

edit: if you know that the csv doesn't contain tabs, you can omit some flags for a more concise

    awk -F, -v OFS='\t' '{print $4+$7,$0}'  foo.csv | sort -n -k1 | cut -f2-
since sort and cut default to tabs/whitespace as separators. If you're unsure about the contents of the CSV, you really need a proper CSV parser.

Re: Become Shell Literate

#308

Post by renox »

voidhorse wrote: post #238

I used to think this way as well, but out of necessity, had to work specifically with a commercial IDE for some time. Turns out, if you are confident learning the keybindings of a robust IDE is worthwhile (e.g. you know that you must use it for some particular project for a decent amount of time) the investment pays off just as well as learning shell commands. A good IDE can do everything a cobbled together shell pip…

The thing is sometimes it's faster to goes to the shell use a grep -r/ag than to wait that 'jump to definition' does its job.. Depends of course the language and the project, but it's true for me at my work big C++ project.. CLion is awfully slow to start, VSCode starts fast but often I can find things faster in the shell than in the IDE :-( so I use it mostly as a dumb text editor.

Re: Become Shell Literate

#309

Post by antb123 »

cjohnson318 wrote: post #257
voidhorse wrote: post #238

I used to think this way as well, but out of necessity, had to work specifically with a commercial IDE for some time. Turns out, if you are confident learning the keybindings of a robust IDE is worthwhile (e.g. you know that you must use it for some particular project for a decent amount of time) the investment pays off just as well as learning shell commands. A good IDE can do everything a cobbled together shell pip…

If you work with mechanical or electrical engineers, then you're going to run into Windows machines pretty quick, and then you need to convince them to install minGW to run your shell scripts. Sure, shell scripts are portable between Mac and *nix, but you're leaving out a large elephant.

Ironically 2 years ago they would have been running unix workstations and due to 2 or two software vendors they have been forced onto windows.

Architects also can use Mac's only part of the time. A lot of software just isn't available for anything other than windows (Rhino etc).

Re: Become Shell Literate

#310

Post by enriquto »

em500 wrote: post #307
enriquto wrote: post #301

Earlier quoted context omitted.

I think this code does not fulfill the task: you need to sum columns 4 and 7, and keep the rest of the data intact in the output.

You're right, that should be awk -F, -v OFS=, '{print $4+$7, $0}' foo.csv | sort -n -k1 -t, | cut -f2- -d, Explanation: awk -F, -v OFS=, sets the input and output column separator to comma, '{print $4+$7, $0}' outputs the sum of column 4 and 7 before the rest of the line. sort -n -k1 -t, sorts the file numerically on column 1, with comma separator. cut -f2- -d, removes column 1, with comma separator. This is of cours…

At the end, it doesn't look that bad. Of course using the csv format is a bad start in unix. Much better to convert everything to tsv and work from there. In that case the "obvious" shell solution is quite clear.
Post reply on HN