Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Finally Bitcasa CEO Explains How The Encryption Works

techcrunch.com

Post reply on HN
Print view View original

21–30 of 42 posts

Re: Finally Bitcasa CEO Explains How The Encryption Works

#21

Post by nikcub »

My biggest issue (beside the initial TC article being a complete shocker) was the claim of 60% saving on de-duplication and that each user only had 25GB of unique data.

This research paper from Microsoft on Farsite[2] claims 'up to 50%' saving on de-dupe with a convergent file system - but that was tested against 500 computers in a corporate environment and it was done back in 2002.

Users now store a lot more photos, a lot more of their own video, and any content that is DRM'd is also unique. You can save on operating system and application files, but it isn't 60%.

There is nothing 'finally' about this additional information. The discussion and criticism of the claims on Twitter was knowing this information about convergent encryption and the key being derived from the content. There is a lot more that is still unanswered - such as how an 'intelligent cache' allows 'unlimited' storage to be available offline.

I really wish these guys would release a research paper with their results, or include more information on their website before they make such bold claims in public.

[1] http://research.microsoft.com/apps/pubs/default.aspx?id=6995...

Re: Finally Bitcasa CEO Explains How The Encryption Works

#22

Post by gojomo »

joshu wrote: post #13

Why is dedupe so important? I have to imagine this mostly helps with OS files that are standard across man machines. Can't we ship a list of hashed client-side?

Dedup also helps with big media files that are identical across many users.

I'm pretty sure Bitcasa has said they avoid uploading known duplicates at all, which suggests the hash (either pre- or post- encryption) is shipped up first.

Re: Finally Bitcasa CEO Explains How The Encryption Works

#23

Post by SODaniel »

joshu wrote: post #13

Why is dedupe so important? I have to imagine this mostly helps with OS files that are standard across man machines. Can't we ship a list of hashed client-side?

Cross account deduplication has nothing to do with making improvement client side, it's all about keeping storage costs down for the provider.

The only type of deduplication that matters to consumers is account specific deduplication, and that saves no data for the provider unless you charge for non deduplicated storage.

Re: Finally Bitcasa CEO Explains How The Encryption Works

#24

Post by SODaniel »

rednaught wrote: post #18
nextparadigms wrote: post #3

"HP: What do you do in terms of encryption or security? TG: We encrypt everything on the client side. We use AES-256 hash, SHA-256 hashing for all the data. HP: So it’s encrypted all on the client side and you can’t look at it on the server side? TG: Exactly" Finally, a company that gets it. I've been asking for this for a while now. I wish Dropbox and all the others would do this, too. I get it that some of Dropbox'…

In addition to Wuala, Spideroak does this as well. A problem remains "with full security" in that you have no idea what's going on in the binary client program. Reveal or open-source the client program and allow customers who need this end-to-end security to compile the program themselves.

We at SpiderOak in fact do not cross-account deduplicate AT ALL and provide a full zero-knowledge environment with no access to client side encryption key info.

We feel that the possible cost savings involved with deduplicating data across user accounts is just not worth the inherent security risks.

Re: Finally Bitcasa CEO Explains How The Encryption Works

#25

Post by trotsky »

joshu wrote: post #13

Why is dedupe so important? I have to imagine this mostly helps with OS files that are standard across man machines. Can't we ship a list of hashed client-side?

the biggest win by far is media files and other large assets like game files or large data sets that many, many people will have duplicate copies of. Even with a pretty low installed base 98%-100% of people's iTunes storage will be deduped. OS files are a big win too, really very very little (by block volume) on most people's drives is unique.

Re: Finally Bitcasa CEO Explains How The Encryption Works

#26

Post by jpluscplusm »

joshu wrote: post #13

Why is dedupe so important? I have to imagine this mostly helps with OS files that are standard across man machines. Can't we ship a list of hashed client-side?

Per-block dedupe, I bet, so it'll get benefits at sub-file levels.

Though given that the vast majority (by volume) of customer data will be people's illegal downloads, their scheme effectively reduces to being a copy of usenet.

Re: Finally Bitcasa CEO Explains How The Encryption Works

#27

Post by JoachimSchipper »

callahad wrote: post #7

Something is twitching in the back of my mind about this. Sure, they can't look at the data based solely on the encrypted copy, but if they have a plaintext copy of a document of interest, they are able to determine which of their customers has that document, right? Doesn't that diminish some of the privacy claims?

Sure, but known-plaintext attacks are not the worst part. Consider this [found via http://www.mail-archive.com/cryptography@metzdowd.com/msg089...]: I take the standard Wordpress config.php [for your host], fill in your site and account name, fill in the one million most common database passwords, and ask the cloud provider whether any of these hashes exist.

Or: I create a form (say .doc) with a single field, CC#, and hope people store this. I then check the existence of 10^11 hashes to find (all customers'!) credit card numbers (for a specific issuer). This takes only a CPU-day! (The network is obviously slower.)

Re: Finally Bitcasa CEO Explains How The Encryption Works

#28

Post by SkyMarshal »

nextparadigms wrote: post #3

"HP: What do you do in terms of encryption or security? TG: We encrypt everything on the client side. We use AES-256 hash, SHA-256 hashing for all the data. HP: So it’s encrypted all on the client side and you can’t look at it on the server side? TG: Exactly" Finally, a company that gets it. I've been asking for this for a while now. I wish Dropbox and all the others would do this, too. I get it that some of Dropbox'…

SpiderOak does this.

https://spideroak.com/engineering_matters

Disclaimer: no affiliation, just like the product. I use SpiderOak to backup private things like AWS keys, KeePass data file, Bitcoin wallet, etc., and Dropbox for documents, photos, and everything else not quite as sensitive.

Re: Finally Bitcasa CEO Explains How The Encryption Works

#29

Post by trotsky »

nikcub wrote: post #21

My biggest issue (beside the initial TC article being a complete shocker) was the claim of 60% saving on de-duplication and that each user only had 25GB of unique data. This research paper from Microsoft on Farsite[2] claims 'up to 50%' saving on de-dupe with a convergent file system - but that was tested against 500 computers in a corporate environment and it was done back in 2002. Users now store a lot more photos,…

any content that is DRM'd is also unique.

In most cases this isn't true. The computation involved in keying media on the fly while it's being downloaded is not insignificant when considered in volume. The added pain of storing everyone's unique keys also discourages this behavior. At worst you'll see different keys being used by region or datacenter, or perhaps key rotation on a weeks-months scale.

Some media (both DRM and non-DRM) will be trivially unique because of metadata like purchaser info or music tags. In some cases this makes the first block unique but all later blocks are deduped, in other cases you need to be somewhat content aware so you can treat the header data separate from the real media data. This also allows you to catch a lot of data people ripped themselves using standard settings.

You can save on operating system and application files, but it isn't 60%.

While I agree that photos and videos will be the bulk of their problem, I don't think that ruins their premise. The question is if their userbase will be significantly overweight on heavy media creators. If it's a standard distribution, I wouldn't be surprised if a majority of people were under 10gb unique and 70%+ deduped.

Re: Finally Bitcasa CEO Explains How The Encryption Works

#30

Post by SODaniel »

nikcub wrote: post #21

My biggest issue (beside the initial TC article being a complete shocker) was the claim of 60% saving on de-duplication and that each user only had 25GB of unique data. This research paper from Microsoft on Farsite[2] claims 'up to 50%' saving on de-dupe with a convergent file system - but that was tested against 500 computers in a corporate environment and it was done back in 2002. Users now store a lot more photos,…

I have to say that I am really curious as to why you would want to base your business viability on 'assumptions' regarding the average amount of users unique data.

I think one missed point when doing this generalization is also the target audience in early adopters for an unlimited service.

As we at SpiderOak quickly found it there are massive differences between 'common users' and the early adopters of a technically proficient service.

I am worried that Bitcasa will be in for a pretty rude awakening in that a large percentage of their early paying users will be JUST the people that store a TON of unique data (Raw files, encrypted data etc), skyrocketing their storage costs in a vulnerable phase of their development.

Post reply on HN