Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Cult of DD

eklitzke.org

Post reply on HN
Print view View original

151–160 of 178 posts

Re: The Cult of DD

#151

Post by heinrich5991 »

amelius wrote: post #135
j3097736 wrote: post #133

Earlier quoted context omitted.

>pv /dev/sdb Huh? pv can cat stuff on it's own, and it will be able to make a progress bar based on the filesize pv image.img > /dev/sdb

So I suppose that the command pv /dev/sdb would actually need to buffer the complete file, before commencing to write to the device (and showing the progress), which would defeat the whole idea of showing progress.

If `pv` doesn't know the input size, it doesn't show it. In your case, it can determine that its stdin is a file by looking at the `/proc/self/fd/0` symlink:

   $ ls -l /proc/self/fd/0  /proc/self/fd/0 -> /tmp/x

Re: The Cult of DD

#152

Post by yjftsjthsd-h »

emmelaich wrote: post #33
merlincorey wrote: post #30

Earlier quoted context omitted.

This is a good point as well. On BSD, we don't have `pv`, but we do have ^T. This will print some sort of status for just about any long running process. It prints very specialized status for certain programs aware of it.

Nice! MacOS has it too (unsurprisingly) Linux, not. I wonder why.

You're not the first to wonder: https://unix.stackexchange.com/questions/179481/siginfo-on-g...

But it looks like the answer is just "it was complicated to implement so Linux didn't add it."

Re: The Cult of DD

#153

Post by tankenmate »

mkup wrote: post #127
tankenmate wrote: post #117

Earlier quoted context omitted.

The other reason for dd existence is that back in the day if you wrote to a block device with a write(2) size of anything other than a multiple of the devices actual block size you'd get a EIO or a EINVAL. So the program you used to make sure the write(2) size was always correct was dd. In the old days some truly weird block devices would accept a write(2) that used only the exact block size, i.e. you could only read…

> back in the day if you wrote to a block device with a write(2) size of anything other than a multiple of the devices actual block size you'd get a EIO or a EINVAL It's not back in the day, it's still true. In Linux, block devices are kernel-cached by default, unless opened with O_DIRECT flag. In general UNIX case (for example, FreeBSD), they aren't: https://www.freebsd.org/doc/en/books/arch-handbook/driverbas... So…

Yes you are right, there are some UNIXes that ship today without a buffer (block) cache and/or don't default to using the block cache when open(2)ing a block device.

For those wondering Linux uses a unified buffer / page cache so there isn't a coherency issue. The buffer cache entries typically point to the corresponding entry in the page cache if it exists. The biggest reason the two are separate but correlated is that the block size isn't always the same as the page size.

Re: The Cult of DD

#154

Post by kazinator »

dd precisely controls the sizes of read, write and lseek system calls. This doesn't matter on buffered block devices; there is no "reblocking" benefit.

Some kinds of devices are structured such that each write produces a discrete block, with a maximum size (such that any bytes in excess are discarded) and each read reads only from one block, advancing to the next one (such that any unread bytes in the current block due to the buffer being too small are discarded). This is very reminiscent of datagram sockets in the IPC/networking arena. dd was developed as an invaluable tool for "reblocking" data for these kinds of devices.

One point that the blog author doesn't realize (or neglects to comment upon) is that "head -c 100MB" relies on an extension, whereas "dd if=/dev/zero of=image.iso bs=4MB count=25" is ... almost POSIX: there is no MB suffix documented by POSIX, only "b" and "k" (lower case). The operator "x" is in POSIX: bs=4x1024x1024.

Here is a non-useless use of dd to request exactly one byte of input from a TTY in raw mode:

file:///usr/share/doc/bash-doc/examples/scripts/line-input.bash

Wrote that myself, back in 1996; was surprised years later to find it in the Bash distribution.

Re: The Cult of DD

#155

Post by gkya »

deno wrote: post #121
gkya wrote: post #107

Earlier quoted context omitted.

I don't know if this works but I believe it doesn't.

It doesn’t. In fact, I would be very careful using subshellsm in general, because it can lead to bugs like this one: ​ http://danwalsh.livejournal.com/74642.html​ .

Well that's strictly a bug caused by mistaken use, as strings are not expanded lazily and heredocs are just another string syntax. How can one use the unix shell without string interpolation? Also, a similar programme would give the wrong result in, say, Ruby or Perl too.

Re: The Cult of DD

#156

Post by stirner »

alayne wrote: post #131
wwalexander wrote: post #130

This article is full of Useless Uses of Cat[1] that could just use redirection operators. For instance, cat image.iso | pv >/dev/sdb could be rewritten as pv /dev/sdb A related mistake is the Useless Use of Echo, since any command of the form echo "foo" | bar can be written using here strings as bar or even bar [1] http://porkmail.org/era/unix/award.html

Useless or not, I personally prefer left to right flow.

You can just use

    /dev/sdb

Re: The Cult of DD

#157

Post by dozzie »

farhaven wrote: post #144
dozzie wrote: post #128

Earlier quoted context omitted.

Erm... It's the first time I see SIGINFO signal. I think you meant SIGUSR1.

On Unix, there's SIGINFO, which doesn't exist on Linux systems. That's why coreutils' dd uses SIGUSR1 instead.

OK, I take I didn't know of BSD-specific signal. But please don't claim it's on Unix, because it's not. Unless you can point me where in SUS (or any other specification) it is defined, as I couldn't find it here: http://pubs.opengroup.org/onlinepubs/009695399/basedefs/sign...

Re: The Cult of DD

#158

Post by gens »

The Ignorance Of Err Ignorant People

dd is a tool. dd can do a lot more then cat. dd can count, seek, skip (seek/drop input), and do basic-ish data conversion. dd is standard, even more standard then cat (the GNU breed). I even used it to flip a byte in a binary, a couple of times.

New-ish gnu dd even adds a nice progress display option (standard is sending it sigusr1, since dd is made to be scripted where only the exit code matters).

> Actually, using dd is almost never necessary, and due to its highly nonstandard syntax is usually just an easy way to mess things up.

Personally I never messed it up, nor was confused about it. This sentence also sets the tone of the whole article, a rather subjective tone that is.

edit: Some dd usage examples: http://www.linuxquestions.org/questions/linux-newbie-8/learn...

Re: The Cult of DD

#159

Post by majewsky »

thom_nic wrote: post #46
viraptor wrote: post #25

There's one good (?) reason to use dd with devices: it specifies target in the same command. For devices, writing to them usually requires root privileges, so it's easy to: sudo dd .... of=/dev/... But there's no trivial cat equivalent: sudo cat ... > target Will open target as your current user anyway. You can play around with tee and redirection of course. But that's getting more complicated than the original.

This. The alternative: sudo sh -c 'cat some.img > /dev/sdb' or even more baroque: cat some.img | sudo tee /dev/sdb > /dev/null is a pain by comparison, and the `sudo sh -c` variant has env implications when spawning a sub-shell. I have an ARM/linux installer script that writes the u-boot image to a specific offset before the first partition: dd if=${UBOOT_DIR}/MLO of=$LO_DEVICE count=1 seek=1 bs=128k dd if=${UBOOT_DI…

Since we're sharing shell tricks: The "sudo tee > /dev/null" may be baroque, but I find it useful whenever I start editing stuff in /etc in vim, only to find that I cannot write my changes because I'm not root. In that case,

  :w !sudo tee %
does the trick. (What "w!" does is send the buffer into the given shell command as stdin.)

Re: The Cult of DD

#160

Post by silky »

xelxebar wrote: post #69
empath75 wrote: post #45

Earlier quoted context omitted.

I'm a Linux sysadmin/developer and I've literally never used a Linux GUI.

I want to do this so bad, but the two things that keep me running X are my browser and mpv. Oh, and viewing PDFs.

Not entirely sure but I think mpv works without X like mplayer.
Post reply on HN