81–90 of 108 posts
I've been using Ubuntu for about a year now and although I feel comfortable doing a lot of things with the CL, I'm not sure if I really know enough about *nix. I wish there was a was a website with the 20-30 most useful unix commands and very clear language as to what they do with examples. Although, I've used all the tools in this post, I still enjoyed the use of example.
One of my favorite little tools that makes all these others better is pv -- PipeViewer. Use it any place in a pipeline to see a progress meter on stderr. Very handy when grepping through a bunch of big log files looking for stuff. Here is a quick strawman example: pv /data/*.log.gz | zgrep -c 'hello world' 241MiB 0:00:15 [15.8MiB/s] [==> ] 2% ETA 0:12:12
BSD has a progress meter utility. It's called progress(1). progress -zf /data/*.log.gz grep -c 'hello world' progress -f /data/*.log.gz zgrep -c 'hello world' The second form will show the progress of the decompression process. You can also adjust buffer size, set the length for the time estimate (otherwise we have to fstat the input), and display progress to stderr instead of stdout.
Thankfully you can also write a Perl one liner. Which most of the times is far powerful than awk.
Earlier quoted context omitted.
Agree. Or actually I start with a 'head -100' so I don't handle too much data in my pipeline until it's ready.
I'm old fashioned, so use "sed 100q" instead of the newer "head -100". It saves a keystroke, too. There are enough variations in ways to do things on Unix that I've sometimes wondered about how easy it would be to identify a user by seeing how they accomplish a common task. For instance, I noticed at one place I worked that even though everyone used the same set of options when doing a "cpio -p", everyone had their o…
So one day I telnetted into a Solaris machine and immediately typed "lsl" before doing anything else. A short while later a colleague came to my cube. He had been snooping the hme1 interface and saw me login. He didn't need to trace the IP because he knew it was me when he saw 3 telnet packets with "l" "s" "l" in them.
Earlier quoted context omitted.
BSD has a progress meter utility. It's called progress(1). progress -zf /data/*.log.gz grep -c 'hello world' progress -f /data/*.log.gz zgrep -c 'hello world' The second form will show the progress of the decompression process. You can also adjust buffer size, set the length for the time estimate (otherwise we have to fstat the input), and display progress to stderr instead of stdout.
Which BSD has that? I don't seem to find it in my FreeBSD installs.
http://svnweb.freebsd.org/base/vendor/tnftp/dist/src/progressbar.c
http://ftp.netbsd.org/pub/NetBSD/NetBSD-release-6/src/usr.bin/{Makefile,progress.c}
Actually useful data science tips for unix users. Make all your commands 3x faster: export LC_ALL=C Actually use the 32 CPUs you paid for: sort --parallel=32 ... xargs -P32 ...
That's part of why Solaris continues to use them in favour of GNU alternatives (although the GNU alternatives are available easily in /usr/gnu/bin).
All you need... is the sum of all values in one particular column."
In that case, if speed was paramount, I'd use Kona or kdb. Unquestionably, k is the best tool for that particular job.
A commenter on the article pointed out the "Useless use of cat". What most users probably don't realize is that the redirection can be anywhere on the line, not just at the beginning. Putting an input redirection at the beginning of the command can make the data flow clearer: from the input file, through the command, to stdout: (This only works for simple commands; you can't do `< file if blah; then foo; else bar; fi…
"Useless use of cat" is one of those boring pedantic comments that makes me cringe. Who cares? It's usually much more straightforward to build a pipeline from left to right, particularly for people who are just learning this stuff.
True, however, people pointing out UUOC are in fact pointing out that you should not be building a pipeline at all. If you want to apply an awk / sed / wc / whatever command to a file, then you should just do that instead of piping it through a extraneous command.
Sure, as people always mention, in your actual workflow you might have a cat or grep already, and are building a pipeline incrementally; there's no reason to remove previous stuff to be "pure" or whatever. But if you're giving a canonical example, there's no reason to add unneeded commands.
I've been using Ubuntu for about a year now and although I feel comfortable doing a lot of things with the CL, I'm not sure if I really know enough about *nix. I wish there was a was a website with the 20-30 most useful unix commands and very clear language as to what they do with examples. Although, I've used all the tools in this post, I still enjoyed the use of example.
Software Carpentry provides a good overview with examples: http://software-carpentry.org/4_0/shell/index.html