Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unix Commands I Wish I’d Discovered Years Earlier

spin.atomicobject.com

Post reply on HN
Print view View original

151–160 of 259 posts

Re: Unix Commands I Wish I’d Discovered Years Earlier

#151

Post by res0nat0r »

ck2 wrote: post #148
teddyh wrote: post #84

Earlier quoted context omitted.

I think you are confused — at no time are any files passed as arguments in the command "find . | wc -l". Also, to count the number of files in a directory, you need to do readdir(), which will get you the names. Then, to see if any of them are directories to be recursed into you need to stat() the names individually. This is all "find" does. How , I'd very much like to know, is this "a nasty hack"?

I am not saying find is outputting files, I am saying it is decoding the full table entry for each file and formatting it, then passing that listing as an entry to wc. Which has to be a bunch of overhead that adds up when you have hundreds of thousands of files to count. All we want from find is "count the number of inode entries in this tree branch that are type f". No formatting, no output, just count. That has to…

You are going to need to rely on the filesystem authors to write a tool to traverse their inode trees directly, or probably write something like that yourself.

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux....

Re: Unix Commands I Wish I’d Discovered Years Earlier

#153

Post by webreac »

dmckeon wrote: post #97
jfb wrote: post #58

Earlier quoted context omitted.

The single most useful thing about ssh to me is that it connects stdout/in across the channel (this is behavior inherited from rsh). This allows for e.g.: % (cd /foo; tar cpf - .) | ssh bar \(cd /baz\; tar xpf -\) Or: % cat ~/.ssh/id_rsa.pub | ssh bar tee -a .ssh/authorized_keys

This is a bit of a pet peeve of mine, but cd/tar should likely be joined with && - in other words, in what directory do you want tar to (not) run if cd fails? So consider: % (cd /foo && tar cpf - .) | ssh bar '(cd /baz && tar xpf -)' or explore something like: % rsync -aSHx --update --delete -e ssh /foo/ user@bar:baz/.

GNU tar has evolved and can use simpler syntax: % tar cp -C /foo . | ssh bar tar xp -C /baz

Re: Unix Commands I Wish I’d Discovered Years Earlier

#154

Post by vectorpush »

ilikepi wrote: post #140
vectorpush wrote: post #130

Command line globbing! For the uninitiated: Let's say my pwd is `/home/projects` and i want to edit `/home/projects/a_huge_sprawling_app/940j_394/lol/flight_controller.rb` `vim **/flight_controller.rb` opens our flight_controller.rb straight away. In terms of effort, this allows you to basically omit a `find -name` when you're in a rush to edit some damn file where the hell is it again damn we need to reevaluate this…

FWIW, the double-star recursive globbing was added in Bash 4.0. My reading of the manual[1] suggests it is disabled by default, and can be enabled by setting the 'globstar' shell option. I believe it's enabled by default in zsh. [1]: https://www.gnu.org/software/bash/manual/bash.html#The-Shopt...

Thanks. Sometimes I forget that zsh is not what everyone uses. I should add zsh to the list, hah!

Re: Unix Commands I Wish I’d Discovered Years Earlier

#155

Post by wiredfool »

WestCoastJustin wrote: post #47
Kurtz79 wrote: post #39

I know it's not exactly an unknown command, but I didn't know about "sort" until last week. It's freaking fast and convenient, sorts hadoop reduce results like a champ.

One of my favorite things about unix is that you can chain commands together with pipe (|). Say for example that you have a example data file that looks like this: # Deflection Col-Force Beam-Force 0.000 0 0 0.001 104 51 0.002 202 101 0.003 298 148 0.0031 104 149 0.004 289 201 0.0041 291 209 0.005 104 250 0.010 311 260 0.020 104 240 You can chain commands together to quickly get ball park figures. I do this all the t…

And once you've done that, join is a handy little utility to do sql style joins on your text files.

Re: Unix Commands I Wish I’d Discovered Years Earlier

#156

Post by jlgreco »

prodigal_erik wrote: post #147
jlgreco wrote: post #118

Earlier quoted context omitted.

I find tmux to be a bit more pleasant to work with. In the past it didn't do reflowing upon resize, but it does that now.

I lean towards nohup time whatever >whatever.out & because so many programs behave differently when attached to a tty (like prompting for input) and I just want a plain text log that says when it succeeded or where it blew up.

Well that is fine, but that is only covering a very small sliver of what screen/tmux are used for.

Re: Unix Commands I Wish I’d Discovered Years Earlier

#158

Post by wiredfool »

ck2 wrote: post #148
teddyh wrote: post #84

Earlier quoted context omitted.

I think you are confused — at no time are any files passed as arguments in the command "find . | wc -l". Also, to count the number of files in a directory, you need to do readdir(), which will get you the names. Then, to see if any of them are directories to be recursed into you need to stat() the names individually. This is all "find" does. How , I'd very much like to know, is this "a nasty hack"?

I am not saying find is outputting files, I am saying it is decoding the full table entry for each file and formatting it, then passing that listing as an entry to wc. Which has to be a bunch of overhead that adds up when you have hundreds of thousands of files to count. All we want from find is "count the number of inode entries in this tree branch that are type f". No formatting, no output, just count. That has to…

OMM, looking at 700k files on a spinning disk, I'm getting an uncached run at 1m13s, and a cached run at 1.2 seconds.

That's "time find . | wc -l"

It's going to be disk limited unless you're in cache.

Switching to "time find . -printf '1' | wc -c" lowers overhead to .7 seconds, so roughly 1us/file.

Re: Unix Commands I Wish I’d Discovered Years Earlier

#160

Post by 9999 »

happywolf wrote: post #4

I recommend Quicksilver on mac, free and powerful. It lets you to find a lot of stuff (application, contact, email, etc.) and manipulate them (send, forward, print, etc.) http://qsapp.com/

I second the Quicksilver recommendation. When you combine Quicksilver triggers (keyboard shortcut available regardless of what application you're in) with applescript, and fabric it's very powerful. I have a trigger that that restarts all of my local services using fabric and opens up new terminal windows running tails of the relevant logs.

The Applescript is this if anyone wants to do something similar:

tell application "Terminal" do script "tail -F /somepath/some.log" do script "tail -F /anotherpath/another.log" do script "sudo fab -f '/pathto/fabfile.py' local deploy" activate end tell

Here's a detailed explanation of setting up Triggers:

http://leafraker.com/2007/09/17/how-to-create-a-quicksilver-...

Post reply on HN