Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unix tricks

mmb.pcb.ub.es

Post reply on HN
Print view View original

31–40 of 232 posts

Re: Unix tricks

#31

Post by niggler »

JoeAcchino wrote: post #21
niggler wrote: post #5

1) `pgrep` is a standard utility that does what his `psgrep` does and much much more. 2) htop is a cpu and memory hog -- every time I've used it I noticed it takes 6+% CPU time 3) there's an awk trick to do the `sort | uniq` recommendation that works on 10+GB files (single pass): awk '!x[$0]++' 4) Passwordless keys are dangerous -- use ssh-agent to save the password of the keys

3) if all the lines of the 10+GB file are actually unique, wouldn't awk keep the whole file in RAM? For files larger than my RAM could this leave my system unresponsive because it's thrashing on swap?

The sort | uniq method literally needs to sort the file and pipe it to uniq, a far more memory-intensive operation than the single-pass awk check. You can write your own hash function in AWK if you think you may overstep memory, but of course you risk hash collisions. It's a tradeoff.

I tried it on a 1U server with 24GB ram a few years ago and found that the sort was thrashing at the 10GB file size while AWK handled it easily

Re: Unix tricks

#32

Post by jfb »

niggler wrote: post #14
carlesfe wrote: post #10

Earlier quoted context omitted.

Not trying to contradict you, just some explanations 1. I prefer 'psgrep' because it covers 99% of my use cases for pgrep (ps axuf | grep $NAME) 2. htop is very nice, come on! Would not let it running on the background for hours, but it's nicer than top 3. Note taken, thanks! 4. Is ssh-agent really safer than using passwordless keys? Just asking, I'm curious

1. `man pgrep` is your friend (you save two greps, and TBH the `grep -v grep` should be a hint that there's a better way) 2. In my experience on Debian (granted this was in 2010), there is a noticeable performance difference between `htop` and `top`. 4. ssh-agent stores the password in memory and is erased on reboot. OTOH If you use a passwordless key file, anyone can use it if they have the key.

1. I always went with

  % ps auwx | grep '[f]oo' | awk '{ print $2; }'

Re: Unix tricks

#33

Post by carlesfe »

niggler wrote: post #31
JoeAcchino wrote: post #21

Earlier quoted context omitted.

3) if all the lines of the 10+GB file are actually unique, wouldn't awk keep the whole file in RAM? For files larger than my RAM could this leave my system unresponsive because it's thrashing on swap?

The sort | uniq method literally needs to sort the file and pipe it to uniq, a far more memory-intensive operation than the single-pass awk check. You can write your own hash function in AWK if you think you may overstep memory, but of course you risk hash collisions. It's a tradeoff. I tried it on a 1U server with 24GB ram a few years ago and found that the sort was thrashing at the 10GB file size while AWK handled…

Yes, awk does some real black bagic. It's awesome how it can parse really, really big files.

Re: Unix tricks

#34

Post by EwanToo »

carlesfe wrote: post #16
EwanToo wrote: post #12

Earlier quoted context omitted.

NFS has 2 mount types, soft and hard. http://tldp.org/HOWTO/NFS-HOWTO/client.html Soft mounts report errors immediately, hard mounts hang. And for permissions, NFS provides everything under the sun that you could possibly need via ACLs. NFSv4 is a very modern protocol, much as SMBv2 is (not SMB though, it's awful). Linux has had NFSv4 for what, 6 years now, at least, and even v2 and v3 had some limited ACL support? h…

That's nice to know, it seems that we have the "hard" configuration at the lab and it's a real pain. Backwards compatibility, you know. For my mini-cluster we use SMB and couldn't be happier.

Not sure what your lab's trying to be backwards compatible with (NFS has had those mount options since at least 1989), but whatever works for you :)

It's a mount option on the client, not the server, so maybe you can change it yourself.

http://www.ietf.org/rfc/rfc1094.txt

Re: Unix tricks

#37

Post by firebones »

Wanted: a lint for your history that analyzes your commandline usage, suggesting these types of tips based on your historical use.

  history | commandlint

Re: Unix tricks

#38

Post by yan »

crusso wrote: post #20

The special bash command I'm most often asked about by shoulder surfers is !$. It substitutes the last argument in the previous command into the current one. For example: $ ls /some/long/path/somewhere/looking/around/ $ cd !$ cd /some/long/path/somewhere/looking/around/

I find ESC, . (esc, then press period) to be more intuitive. (alt+period works as well)

Re: Unix tricks

#39

Post by kps »

crusso wrote: post #20

The special bash command I'm most often asked about by shoulder surfers is !$. It substitutes the last argument in the previous command into the current one. For example: $ ls /some/long/path/somewhere/looking/around/ $ cd !$ cd /some/long/path/somewhere/looking/around/

An arguably better (and slightly more portable) way to accomplish this is the special variable $_ , which expands to the last argument of the previous command. Since $_ is a variable rather than a history substitution, it still works when there is no command history (e.g. in scripts) and allows all the usual variable expansion forms, for example ${_##*/} to extract the last path component.

Re: Unix tricks

#40

Post by ominous_prime »

carlesfe wrote: post #10
niggler wrote: post #5

1) `pgrep` is a standard utility that does what his `psgrep` does and much much more. 2) htop is a cpu and memory hog -- every time I've used it I noticed it takes 6+% CPU time 3) there's an awk trick to do the `sort | uniq` recommendation that works on 10+GB files (single pass): awk '!x[$0]++' 4) Passwordless keys are dangerous -- use ssh-agent to save the password of the keys

Not trying to contradict you, just some explanations 1. I prefer 'psgrep' because it covers 99% of my use cases for pgrep (ps axuf | grep $NAME) 2. htop is very nice, come on! Would not let it running on the background for hours, but it's nicer than top 3. Note taken, thanks! 4. Is ssh-agent really safer than using passwordless keys? Just asking, I'm curious

re: 4: not only is an ssh agent by far safer, but most agents now allow you to set a timeout on a key, so it's not indefinitely saved in memory.

A passwordless key gives anyone with acces to that file, access to the login associated with it. If that file is inadvertently exposed (oops, checked it into github...), any machine you have a login on must be considered compromised.

Post reply on HN