Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Encrypted, deduplicated remote backups

stavros.io

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Encrypted, deduplicated remote backups

#41

Post by mariusblaesing »

StavrosK wrote: post #39
mariusblaesing wrote: post #35

Earlier quoted context omitted.

You can still tell what exactly happens. Two separate processes might be used: one for crypto + network which is open source, the one manages synchronisation (which is the secret sauce of all cloud-sync services, if it works really good). Updated the FAQ: https://boldshare.com/?lang=en#faq

You won't know if the other process is transmitting your key, though.

Communication between open and closed source processes runs via IPC. Since all the IPC functions are declared in the open source part, you can exactly check what data is exchanged between the processes -> You can see that keys are not transmitted.

Re: Encrypted, deduplicated remote backups

#42

Post by Wilya »

ams6110 wrote: post #34
ColinWright wrote: post #7

What's wrong with tarsnap? http://www.tarsnap.com/ Added in edit: I see that people are saying it's expensive, and I appreciate that. When I bundle it into my considerations of how much I spend on other things, what I get, and my level of confidence in the service, it seems a good deal to me. In the interests of full disclosure, I don't actually use anything like this, as I have a storage solution provided for me. Bu…

Expensive? Don't we have patio11 regularly telling Colin (Percival) that he's not charging nearly enough?

By telling him to charge more, what he's saying is more or less to drop the consumer market and sell to bigger businesses.

It would probably be better for Tarsnap's financial health, but it wouldn't make it a better product for people like the OP who just want to save pictures of their dog.

Re: Encrypted, deduplicated remote backups

#43

Post by parley »

mariusblaesing wrote: post #35
parley wrote: post #12

Earlier quoted context omitted.

> Client code will be open source. On the link you provided I found the following: "Parts of the programm relevant for security and networking actions will be open source." "Parts of the program" might not do it for everyone (me included). Will anyone be able to audit the full source code of the client and compile it themselves, like e.g. tarsnap? Otherwise I fear the usual trust issues arise. If I missed some inform…

You can still tell what exactly happens. Two separate processes might be used: one for crypto + network which is open source, the one manages synchronisation (which is the secret sauce of all cloud-sync services, if it works really good). Updated the FAQ: https://boldshare.com/?lang=en#faq

I think there's plenty of room for the larger discussion about how much we actually (ought to) trust a lot of our closed source software that runs free and non-jailed on our computers day in and day out - and recent events haven't affected that viewpoint very much in either direction. However, data synchronization software gets tougher love from everyone these days for apparent reasons.

I read the updated FAQ and saw the note about blocking network I/O for the closed source process. If the closed source process in your client were locked down in such a way that it was only ever invoked from the open source process and had no way of doing any I/O whatsoever (network or otherwise) except with the open source process (where what goes in and comes out can be audited), then I'd agree this sounds quite interesting.

I'll be waiting as well, and sincerely hope you deliver something nice!

Re: Encrypted, deduplicated remote backups

#44

Post by StavrosK »

mariusblaesing wrote: post #41
StavrosK wrote: post #39

Earlier quoted context omitted.

You won't know if the other process is transmitting your key, though.

Communication between open and closed source processes runs via IPC. Since all the IPC functions are declared in the open source part, you can exactly check what data is exchanged between the processes -> You can see that keys are not transmitted.

How can you check that the closed-source process won't ever read the key from the disk without elaborate contortions?

Re: Encrypted, deduplicated remote backups

#45

Post by _swa8 »

StavrosK wrote: post #27
joeyh wrote: post #24

Earlier quoted context omitted.

obnam's speed is mostly determined by file size and latency. With large files and low latency, it can push several megabytes a second off my laptop. It's pretty excellent in every other respect.

Hmm, I'll give it another shot, thanks. Does it only transfer the diffs too, or does it have to read everything off the remote server?

http://liw.fi/obnam/ondisk/ attempts to answer that in some detail. In summary: Obnam splits file data into chunks and stores each chunk only once. It looks up chunks using checksums, and does not need to download the chunk data again (unless you turn on verification mode to protect against checksum collisions). Obnam does download B-tree nodes, though, but they should be a fraction of the size of the file content data.

Re: Encrypted, deduplicated remote backups

#46

Post by mariusblaesing »

StavrosK wrote: post #44
mariusblaesing wrote: post #41

Earlier quoted context omitted.

Communication between open and closed source processes runs via IPC. Since all the IPC functions are declared in the open source part, you can exactly check what data is exchanged between the processes -> You can see that keys are not transmitted.

How can you check that the closed-source process won't ever read the key from the disk without elaborate contortions?

What should it do with it? It can't send it anywhere: Block network access for that process if you don't trust it.

Key won't be accessible on disk anyway, only in RAM during crypto and will be destroyed immediately afterwards.

Re: Encrypted, deduplicated remote backups

#47

Post by avtar »

nilved wrote: post #21

I feel that the main problem with online backups is simply a matter of bandwidth. I have in excess of 3 TB of data and even if I only backed up things I considered _critical_, it would still be so much that it would take a week to upload and cost me hundreds of dollars of data overages. Such is Canada. It's very telling that I'm even considering this, but if there existed a backup service where I could mail them a ha…

It's possible to mail drives to Amazon and have them import the initial data set http://aws.amazon.com/importexport/ A former employer of mine did that a few years ago so that they could start serving a lot of high resolution images using S3.

Re: Encrypted, deduplicated remote backups

#48

Post by StavrosK »

mariusblaesing wrote: post #46
StavrosK wrote: post #44

Earlier quoted context omitted.

How can you check that the closed-source process won't ever read the key from the disk without elaborate contortions?

What should it do with it? It can't send it anywhere: Block network access for that process if you don't trust it. Key won't be accessible on disk anyway, only in RAM during crypto and will be destroyed immediately afterwards.

You can make it so that it will only store the key in RAM, but then you'd have to enter the key every time it launches, making automatic backups impossible.

Although, I agree, if you only allow the open source network access, AND can ensure that the key won't somehow be smuggled in the data the closed source process sends, you're probably fine.

Re: Encrypted, deduplicated remote backups

#49

Post by StavrosK »

_swa8 wrote: post #45
StavrosK wrote: post #27

Earlier quoted context omitted.

Hmm, I'll give it another shot, thanks. Does it only transfer the diffs too, or does it have to read everything off the remote server?

http://liw.fi/obnam/ondisk/ attempts to answer that in some detail. In summary: Obnam splits file data into chunks and stores each chunk only once. It looks up chunks using checksums, and does not need to download the chunk data again (unless you turn on verification mode to protect against checksum collisions). Obnam does download B-tree nodes, though, but they should be a fraction of the size of the file content da…

Thank you for the summary, it sounds pretty reasonable. I will read the full document now, hopefully it will elucidate even more.

Re: Encrypted, deduplicated remote backups

#50

Post by StavrosK »

jackalope wrote: post #3

EncFS in reverse mode with rdiff-backup would pretty much be ideal, but EncFS currently has two bugs that prevent this from working. Does anyone know what this refers to? It sounds like a winning combination to me.

I just found out about python-fuse, and it seems that I can use a lot of the code from encbup to write a simple EncFS clone that will accept a list of directories and produce a mountpoint that will expose them encrypted. Then you can just rdiff-backup that and you're done.

What does everyone think?

Post reply on HN