Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux utils that you might not know

shiroyasha.io

Post reply on HN
Print view View original

61–70 of 159 posts

Re: Linux utils that you might not know

#61

Post by dom0 »

LukeShu wrote: post #37
Scaevolus wrote: post #28

Earlier quoted context omitted.

No. Ext4 doesn't do data journaling by default. Even when enabled, what's written to the journal is blank data that's about to be written, not the current file's contents. 1) write to journal "I'm going to overwrite this file with this data (zeros)" 2) commit journal 3) write data to file This is typical for journaling filesystems-- step 3 can be interrupted by a crash and replayed later (by re-reading the journal).…

Before evaluating the claims myself: The shred manual specifically claims that it is "not guaranteed to be effective" on "log-structured or journaled file systems", and specifically calls out ext3 in data=journal mode. I would assume that the concern with ext3/4 in data=journal mode is that shred does not guarantee that the records of previous writes are evicted from the journal.

In data=journal mode, data to be written is first written into the journal. Only after the journal is flushed it will be written out to the correct location. Therefore, a crash at any time is fixed by replaying the journal forwards.

Note that the ext3/4 journal is a redo log, not an undo log. Old file contents are not copied into the journal on a write.

Thus, I don't see why shred should be less effective in data=journal mode compared to the other journaling modes.

CoW file systems are a different story. They don't allow you to overwrite physical file contents. You have to set the +C (FL_NOCOW) flag, which is, by principle, only effective for a file that does not have any contents yet. Thus, you can't set +C on an existing file and overwrite it's contents.

Re: Linux utils that you might not know

#62

Post by AstralStorm »

noja wrote: post #36
ivanstojic wrote: post #15

It always pains me when I see people use "cat" left and right, even when they don't need it. This makes for good reading: http://porkmail.org/era/unix/award.html

It shouldn't. It's 2017 now, and the speed difference between a "misplaced" cat command is normally negligible. The speed of the terminal user is more important.

It is slower to type too. Just write the command with the file name already.

Re: Linux utils that you might not know

#63

Post by hawski »

tduk wrote: post #44

Don't forget the often overlooked "apropos"!

In my opinion it sucks. It is possible that I suck, but I almost never can find what I'm searching for with it.

Now I found that full text search is man -K, but it searches through sources so it can also be quite useless. Is there a desktop full text search for man pages? It should use rendered man pages. It could be a fun project. Google doesn't count and I use it already.

Re: Linux utils that you might not know

#64

Post by flukus »

Tepix wrote: post #56
flukus wrote: post #49

Earlier quoted context omitted.

You get a blank screen, but if you do: cat file-that-doesnt-exist.txt | less You get a nice error message: cat: 'file-that-doesnt-exist.txt': No such file or directory

Works for me: $ I just tried this on bash 4.3. Cat remains superfluous in this case.

Just tried it again and you're right it does produce an error, I was sanitizing a real error I just had though. With the real one I had an unescaped space, cat mentioned that it couldn't open "both" files, in just gave me a message about and "ambiguous redirect"

Re: Linux utils that you might not know

#65

Post by cnvogel »

VSpike wrote: post #41
RijilV wrote: post #22

I was really happy about tac when I found out about it, same with paste and comm as already mentioned. Gotten a lot of mileage out of the various seldom used options of uniq, diff and cut too.

I have a friend who uses `tac | tac` in pipelines to make the pipeline wait for all the data before continuing, which is an interesting hack.

If you install the "moreutils", there's sponge doing the same.

https://joeyh.name/code/moreutils/ https://linux.die.net/man/1/sponge

Re: Linux utils that you might not know

#66

Post by kingosticks »

falsedan wrote: post #48
MichaelBurge wrote: post #8

* 'comm -3' is a quick way to do a set difference from the command line. * Newer versions of sort have an option for running sorts in parallel. If you're using an older sort, you can split the files, sort them individually with GNU parallel, and use --merge to combine them. * If you have scripts that read data files, process them, and output more files, consider using a Makefile. * tmux is a good way to leave a devel…

> If you have scripts that read data files, process them, and output more files, consider using a Makefile. Dear sweet hypnotoad, don't use Makefiles for this. Scripts in a pipeline are perfectly well suited for ETL. They have the advantage of using the same language as the command language (Makefile is not shell, and when it differs it's a significant surprise). Plus, you can drop a script into a dir in your PATH an…

I'd stick with pushd and popd every time, it's considerably more expressive.

Re: Linux utils that you might not know

#67

Post by anc84 »

AstralStorm wrote: post #62
noja wrote: post #36

Earlier quoted context omitted.

It shouldn't. It's 2017 now, and the speed difference between a "misplaced" cat command is normally negligible. The speed of the terminal user is more important.

It is slower to type too. Just write the command with the file name already.

If you are just doing one line and you write it perfectly, then maybe. For many people cat is simply the start if excessive piping. I like having the filename as far at the beginning as possible so I can Ctrl-W with ease.

Re: Linux utils that you might not know

#68

Post by Aissen »

Just look at anything in the "moreutils" package. Just great stuff. Excerpt:

- ifdata: do not parse the output of ifconfig/ip anymore. Just use this tool.

- sponge: when you need to overwrite an input file at the end of pipe. sponge will wait for the pipe to end before overwriting, preventing any data loss

- vidir: edit a given dir with your EDITOR. Awesome for mass renames/deletes.

- ts: add timestamps to a command

- parallel: C implementation of GNU parallel (in perl). Very small, very fast, just does the core (running commands in parallel), and does not try to take over xargs.

There are others of course.

Re: Linux utils that you might not know

#69

Post by falsedan »

kingosticks wrote: post #66
falsedan wrote: post #48

Earlier quoted context omitted.

> If you have scripts that read data files, process them, and output more files, consider using a Makefile. Dear sweet hypnotoad, don't use Makefiles for this. Scripts in a pipeline are perfectly well suited for ETL. They have the advantage of using the same language as the command language (Makefile is not shell, and when it differs it's a significant surprise). Plus, you can drop a script into a dir in your PATH an…

I'd stick with pushd and popd every time, it's considerably more expressive.

> considerably more expressive

is that an advantage? Do you have the time to explain this a bit more?

I feel that new users need less expressiveness, to avoid decision overload, and keeping one automatic directory save point is easier to mentally manage than a stack of them.

I do recommend using pushd/popd in shell scripts (always) and interactively (if you must), but I think 'cd -' should be the first thing you introduce to newcomers w.r.t tracking working directory changes.

Post reply on HN