Time Machine and Mail: a match made in hell

#1

Post by lisper » Fri, Jul 31, 2009, 7:17 PM UTC

Time Machine and Mail: a match made in hell
rondam.blogspot.com

Re: Time Machine and Mail: a match made in hell

#2

Post by a2tech » Fri, Jul 31, 2009, 8:33 PM UTC

Creating an Archive folder as he describes doesn't stop Time Machine from creating all those hard links everytime it runs-so his solution doesn't fix anything. He should upgrade to a faster disk for his backup to solve his issues. I have a very large collection of mailboxes on my machine and see none of these issues he's describing.

Re: Time Machine and Mail: a match made in hell

#3

Post by emordnilap » Fri, Jul 31, 2009, 8:53 PM UTC

a2tech wrote: post #2

Creating an Archive folder as he describes doesn't stop Time Machine from creating all those hard links everytime it runs-so his solution doesn't fix anything. He should upgrade to a faster disk for his backup to solve his issues. I have a very large collection of mailboxes on my machine and see none of these issues he's describing.

I think he meant that making a hard link and copying a small file take about the same time (when storing emails as individual files.)

I think the archive folder idea is a workaround for Entourage. If you have one large file that rarely changes, then making a hard link to it would be much faster than copying it.

Re: Time Machine and Mail: a match made in hell

#4

Post by Locke1689 » Fri, Jul 31, 2009, 9:05 PM UTC

And for small files, like most mail messages tend to be unless they have a lot of attachments, creating a hard link is no faster than actually copying the file.

[citation needed]

Re: Time Machine and Mail: a match made in hell

#5

Post by mjtsai » Fri, Jul 31, 2009, 9:12 PM UTC

a2tech wrote: post #2

Creating an Archive folder as he describes doesn't stop Time Machine from creating all those hard links everytime it runs-so his solution doesn't fix anything. He should upgrade to a faster disk for his backup to solve his issues. I have a very large collection of mailboxes on my machine and see none of these issues he's describing.

Creating an archive folder does help because then Time Machine can create one hard link for the directory rather than a folder containing thousands of hard links to individual message files.

The archive folder is for Apple Mail, not Entourage. Entourage stores all the mail in a single database so it doesn't matter which folders the messages are in.

Re: Time Machine and Mail: a match made in hell

#6

Post by jmtulloss » Fri, Jul 31, 2009, 10:08 PM UTC

Locke1689 wrote: post #4

And for small files, like most mail messages tend to be unless they have a lot of attachments, creating a hard link is no faster than actually copying the file. [citation needed]

In my unscientific trial, it looks like the hard link is an order of magnitude faster. Still, it takes time.

  $ du -sh README.markdown
  4.0K	README.markdown

  $ time ln README.markdown README.markdown.link

  real	0m0.002s
  user	0m0.000s
  sys	0m0.001s

  $ time cp README.markdown README.markdown.copy

  real	0m0.041s
  user	0m0.000s
  sys	0m0.002s

Re: Time Machine and Mail: a match made in hell

#7

Post by [deleted] » Fri, Jul 31, 2009, 10:11 PM UTC

Locke1689 wrote: post #4

And for small files, like most mail messages tend to be unless they have a lot of attachments, creating a hard link is no faster than actually copying the file. [citation needed]

[deleted]

Re: Time Machine and Mail: a match made in hell

#8

Post by jff » Fri, Jul 31, 2009, 11:19 PM UTC

The real question is why the hell Time Machine backs up at a file level instead of at a block level.

Plan 9's Venti archival storage (http://en.wikipedia.org/wiki/Venti and http://plan9.bell-labs.com/sys/doc/venti/venti.html) stores data as blocks, which are referenced by SHA1 hashes. Since these blocks range between 512 bytes and 56 KB, a large file (like a mail file) gets split into many blocks. The brilliant thing is that a given block will only ever be stored once; your top-level filesystem then only has to keep track of which SHA1 hashes make up a file.

With this system, people have been keeping daily snapshots of their filesystems over the course of years, and the space consumption rate actually tends to decrease over time--see the graphs at http://plan9.bell-labs.com/sys/doc/venti/venti.html

Re: Time Machine and Mail: a match made in hell

#9

Post by lisper » Fri, Jul 31, 2009, 11:34 PM UTC

jmtulloss wrote: post #6

Locke1689 wrote: post #4

And for small files, like most mail messages tend to be unless they have a lot of attachments, creating a hard link is no faster than actually copying the file. [citation needed]

In my unscientific trial, it looks like the hard link is an order of magnitude faster. Still, it takes time. $ du -sh README.markdown 4.0K README.markdown $ time ln README.markdown README.markdown.link real 0m0.002s user 0m0.000s sys 0m0.001s $ time cp README.markdown README.markdown.copy real 0m0.041s user 0m0.000s sys 0m0.002s

That's pretty unscientific :-) To measure anything that fast reliably you have to do it more than once:

    [ron@mickey:~/foo]$ cat foo
    foo
    
    [ron@mickey:~/foo]$ time for i in {1..1000}; do cp foo foo.c.$i; done
    real	0m2.705s
    user	0m0.524s
    sys	0m2.190s
    
    [ron@mickey:~/foo]$ time for i in {1..1000}; do ln foo foo.l.$i; done
    real	0m2.618s
    user	0m0.486s
    sys	0m2.008s

Re: Time Machine and Mail: a match made in hell

#10

Post by xenophanes » Fri, Jul 31, 2009, 11:42 PM UTC

a2tech wrote: post #2

Creating an Archive folder as he describes doesn't stop Time Machine from creating all those hard links everytime it runs-so his solution doesn't fix anything. He should upgrade to a faster disk for his backup to solve his issues. I have a very large collection of mailboxes on my machine and see none of these issues he's describing.

if nothing in a folder changed, then the individual files aren't hardlinked. if you dump 5k files into an archive folder, then assuming your inbox has an update, and the archive folder doesn't, it's 5k less hardlinks being made.