Wouldn't it be nice if Google just dumped the takeout into a sqlite file?
How would that help 99.5% of their user base?
Incremental Backups of Gmail Takeouts
21–30 of 64 posts
Re: Incremental Backups of Gmail Takeouts
#22Serious question: have you ever needed an email from even 5 years ago? I only save financial statements and contact information. Everything else gets deleted as soon as possible.
It has happened, but it's rare.
I let it pile up, rarely delete anything except marketing emails. Over 30K emails in my gmail inbox.
Re: Incremental Backups of Gmail Takeouts
#23Re: Incremental Backups of Gmail Takeouts
#24> if you want to back this file up regularly with something like restic, then you will quickly end up in a world of pain: since new mails are not even appended to the end of the file, each cycle of takeout-then-backup essentially produces a new giant file. As I'm sure the author is aware, Restic will do hash-based chunking so that similar files can be efficiently be backed up. How similar are two successive Takeout m…
> As I'm sure the author is aware, Restic will do hash-based chunking so that similar files can be efficiently be backed up.
> Even if the order of the mbox's messages are ~random, Restic's delta updates will forego large attachments.
I forget the exact number, but the rolling hashes for Restic and Borg are tuned to produce chunks sizes on the order of an entire megabyte.
Which means attachment file sizes need to be many megabytes in order for Restic to be much use, since the full chunk has to fall within the attachment. — You'd lose 0.5MB at both ends of each attachment on average, so a 5MB file would only be 80% deduped.
Nothing against Restic, but it's tuned for file-level backup, and I'm sure it wouldn't be as performant if it used chunks that were small enough to pick apart individual e-mails.
I suggested the author check out ZPAQ, which has a user-tunable average fragment size, and is arguably even simpler than Restic.
The ZPAQ file can then itself be efficiently backed up by Restic.
Re: Incremental Backups of Gmail Takeouts
#25Wouldn't it be nice if Google just dumped the takeout into a sqlite file?
Instead of the native format for e-mail archives? Supported by every MUA ever made? Not really, no.
Re: Incremental Backups of Gmail Takeouts
#26Have you looked into using a full MIME/mbox parser library, e.g. GMime [0] or MimeKit [1]? Both support parsing mbox files directly, and they should be able to handle the intricacies of parsing any messages/attachments you throw at them. Then you could write out the MIME representation of each message (including any attachments) into its own file and then check for new messages. That way you can be sure each “chunk” represents a single message in its entirety. Not sure if this is any better since your solution seems to work pretty well.
Re: Incremental Backups of Gmail Takeouts
#27Serious question: have you ever needed an email from even 5 years ago? I only save financial statements and contact information. Everything else gets deleted as soon as possible.
When I was moving to Canada I had to dig stuff back from 6 years before!
That's why I keep all of them
Re: Incremental Backups of Gmail Takeouts
#28Serious question: have you ever needed an email from even 5 years ago? I only save financial statements and contact information. Everything else gets deleted as soon as possible.
Not on gmail but a company I worked for sent me my pay slips over email. While I also printed them, I also forwarded them to my private email address and kept them to this day on a separate mbox file.
Also when I can't remember the age of my nephews or the postal address of my siblings I just dig the birth / move announcements in my emails.
Re: Incremental Backups of Gmail Takeouts
#29Isn't it easier to just backup via IMAP to maildir versioned with git?
Does takeout include any metadata not accessible via IMAP? Does it even include labels?
Re: Incremental Backups of Gmail Takeouts
#30Wouldn't it be nice if Google just dumped the takeout into a sqlite file?
absolutely not