Time Machine and Mail: a match made in hell

Post by rarrrrrr » Sat, Aug 01, 2009, 12:46 AM UTC

jff wrote: post #8

The real question is why the hell Time Machine backs up at a file level instead of at a block level. Plan 9's Venti archival storage ( http://en.wikipedia.org/wiki/Venti and http://plan9.bell-labs.com/sys/doc/venti/venti.html ) stores data as blocks, which are referenced by SHA1 hashes. Since these blocks range between 512 bytes and 56 KB, a large file (like a mail file) gets split into many blocks. The brilliant thi…

Post by wmf » Sat, Aug 01, 2009, 1:14 AM UTC

rarrrrrr wrote: post #11

jff wrote: post #8

The real question is why the hell Time Machine backs up at a file level instead of at a block level. Plan 9's Venti archival storage ( http://en.wikipedia.org/wiki/Venti and http://plan9.bell-labs.com/sys/doc/venti/venti.html ) stores data as blocks, which are referenced by SHA1 hashes. Since these blocks range between 512 bytes and 56 KB, a large file (like a mail file) gets split into many blocks. The brilliant thi…

Backup systems make tradeoffs to be space efficient (block level backups) or computationally efficient (using full file backups that have mostly IO cost.) Apple made the choice to have time machine operate with little CPU burden. While this would be a tremendously poor choice for an online storage system like SpiderOak, Dropbox, SugarSync, etc. it probably makes sense for them since it's usually working with a local…

Post by rarrrrrr » Sat, Aug 01, 2009, 5:46 AM UTC

wmf wrote: post #13

rarrrrrr wrote: post #11

Earlier quoted context omitted.

Backup systems make tradeoffs to be space efficient (block level backups) or computationally efficient (using full file backups that have mostly IO cost.) Apple made the choice to have time machine operate with little CPU burden. While this would be a tremendously poor choice for an online storage system like SpiderOak, Dropbox, SugarSync, etc. it probably makes sense for them since it's usually working with a local…

Apple made the choice to have Time Machine operate with little CPU burden. I don't know why, given that my CPU is a sunk cost. (Also, I suspect that Apple chose hard links instead of deltas for ease of implementation, not to save CPU cycles.) last I checked, Venti didn't support recovering the space from deleted items, except by way of making a new copy of the file system. I think the state of the art has moved on si…

Post by allenbrunson » Sat, Aug 01, 2009, 7:16 AM UTC

gaius wrote: post #16

jff wrote: post #8

The real question is why the hell Time Machine backs up at a file level instead of at a block level. Plan 9's Venti archival storage ( http://en.wikipedia.org/wiki/Venti and http://plan9.bell-labs.com/sys/doc/venti/venti.html ) stores data as blocks, which are referenced by SHA1 hashes. Since these blocks range between 512 bytes and 56 KB, a large file (like a mail file) gets split into many blocks. The brilliant thi…

Time Machine can be made to be "aware" of files that are effectively filesystems themselves - you can watch it as it backs up VirtualPC for example. It doesn't take the entire 10G virtual disk image every time.

Post by jrockway » Sat, Aug 01, 2009, 11:58 AM UTC

lisper wrote: post #9

jmtulloss wrote: post #6

Earlier quoted context omitted.

In my unscientific trial, it looks like the hard link is an order of magnitude faster. Still, it takes time. $ du -sh README.markdown 4.0K README.markdown $ time ln README.markdown README.markdown.link real 0m0.002s user 0m0.000s sys 0m0.001s $ time cp README.markdown README.markdown.copy real 0m0.041s user 0m0.000s sys 0m0.002s

That's pretty unscientific :-) To measure anything that fast reliably you have to do it more than once: [ron@mickey:~/foo]$ cat foo foo [ron@mickey:~/foo]$ time for i in {1..1000}; do cp foo foo.c.$i; done real 0m2.705s user 0m0.524s sys 0m2.190s [ron@mickey:~/foo]$ time for i in {1..1000}; do ln foo foo.l.$i; done real 0m2.618s user 0m0.486s sys 0m2.008s