Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Encrypted, deduplicated remote backups

stavros.io

Post reply on HN
Print view View original

21–30 of 50 posts

Re: Encrypted, deduplicated remote backups

#21

Post by nilved »

I feel that the main problem with online backups is simply a matter of bandwidth. I have in excess of 3 TB of data and even if I only backed up things I considered _critical_, it would still be so much that it would take a week to upload and cost me hundreds of dollars of data overages. Such is Canada.

It's very telling that I'm even considering this, but if there existed a backup service where I could mail them a hard drive to load my data initially, and then upload only incremental changes, I would be all over that. Of course, uploading videos and such would still be out of the question.

Re: Encrypted, deduplicated remote backups

#22

Post by purplerails »

Really nice utility. More developers should dip their toes into crypto and develop applications like this. :)

A comment in the code about why it's OK in your case to use the same key for MAC and encrypt would be useful. I think you're fine. See here: http://security.stackexchange.com/questions/37880/why-cant-i...

I needed to implement deduplication in my system. Since I controlled the server, I developed a slightly more elaborate system which doesn't have the limitation of a predictable IV (predictable from the encryption key).

So in my system, I derive two keys from the same passphrase (PBKDF2 with different salts). I encrypt as usual with unpredictable IVs. When uploading, the HMAC of the plaintext and SHA-256 of the ciphertext are both loaded.

To check for duplication, the client asks if a certain HMAC is already present. And it's an error (at the server) to upload multiple ciphertexts with the same HMAC.

Re: Encrypted, deduplicated remote backups

#23

Post by StavrosK »

nilved wrote: post #21

I feel that the main problem with online backups is simply a matter of bandwidth. I have in excess of 3 TB of data and even if I only backed up things I considered _critical_, it would still be so much that it would take a week to upload and cost me hundreds of dollars of data overages. Such is Canada. It's very telling that I'm even considering this, but if there existed a backup service where I could mail them a ha…

Same here, but I can do with a week if it can resume properly and then only upload the diffs. Hopefully, encbup+bup can do that well.

Re: Encrypted, deduplicated remote backups

#24

Post by joeyh »

StavrosK wrote: post #18
skrause wrote: post #11

If you're using a Linux-based system http://liw.fi/obnam/ might also be a good choice.

I tried that on 500 KB when I was testing it and it took a few minutes to back things up. I'm not sure if it's linear, but I wouldn't want to try the full set if it's that slow...

obnam's speed is mostly determined by file size and latency. With large files and low latency, it can push several megabytes a second off my laptop.

It's pretty excellent in every other respect.

Re: Encrypted, deduplicated remote backups

#25

Post by StavrosK »

purplerails wrote: post #22

Really nice utility. More developers should dip their toes into crypto and develop applications like this. :) A comment in the code about why it's OK in your case to use the same key for MAC and encrypt would be useful. I think you're fine. See here: http://security.stackexchange.com/questions/37880/why-cant-i... I needed to implement deduplication in my system. Since I controlled the server, I developed a slightly m…

The vulnerability in that post is for using AES-CBC with AES-CBC-MAC with the same key. I'm using AES-CBC and HMAC-SHA512, which should be okay. The design was reviewed by cryptographers and was given a green light, plus I tried to use as little custom crypto as possible for this exact reason :)

Re: Encrypted, deduplicated remote backups

#26

Post by SimonPStevens »

nilved wrote: post #21

I feel that the main problem with online backups is simply a matter of bandwidth. I have in excess of 3 TB of data and even if I only backed up things I considered _critical_, it would still be so much that it would take a week to upload and cost me hundreds of dollars of data overages. Such is Canada. It's very telling that I'm even considering this, but if there existed a backup service where I could mail them a ha…

"if there existed a backup service where I could mail them a hard drive to load my data initially"

Crashplan provide this option. I've never used that specific part of their service personally, but I've been a satisfied customer of their unlimited family plan for several years.

Re: Encrypted, deduplicated remote backups

#27

Post by StavrosK »

joeyh wrote: post #24
StavrosK wrote: post #18

Earlier quoted context omitted.

I tried that on 500 KB when I was testing it and it took a few minutes to back things up. I'm not sure if it's linear, but I wouldn't want to try the full set if it's that slow...

obnam's speed is mostly determined by file size and latency. With large files and low latency, it can push several megabytes a second off my laptop. It's pretty excellent in every other respect.

Hmm, I'll give it another shot, thanks. Does it only transfer the diffs too, or does it have to read everything off the remote server?

Re: Encrypted, deduplicated remote backups

#28

Post by brightsize »

SimonPStevens wrote: post #26
nilved wrote: post #21

I feel that the main problem with online backups is simply a matter of bandwidth. I have in excess of 3 TB of data and even if I only backed up things I considered _critical_, it would still be so much that it would take a week to upload and cost me hundreds of dollars of data overages. Such is Canada. It's very telling that I'm even considering this, but if there existed a backup service where I could mail them a ha…

"if there existed a backup service where I could mail them a hard drive to load my data initially" Crashplan provide this option. I've never used that specific part of their service personally, but I've been a satisfied customer of their unlimited family plan for several years.

Right. See the "Seeded backup" section here: http://www.crashplan.com/consumer/details.html

Re: Encrypted, deduplicated remote backups

#29

Post by ctz »

I'm disappointed there's no comment here from tptacek telling you off for that decidedly dodgy-looking mac-then-encrypt, or the timing side channel in your mac verification. He must be slipping :)
Post reply on HN