Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Megabad: A quick look at the state of Mega’s encryption

arstechnica.com

Post reply on HN
Print view View original

51–60 of 60 posts

Re: Megabad: A quick look at the state of Mega’s encryption

#51

Post by roc »

jerf wrote: post #37
roc wrote: post #23

Earlier quoted context omitted.

It sounds to me like Mega is just using block-level hardware de-duplication and they're making sure it's clearly spelled out in the Terms. After all, if each file is being encrypted with different keys, then Alice and Bob's encrypted copies of THE HOBBIT wouldn't match at all. So much as removing the file from Mega's servers and re-uploading it would seemingly change the key and thus the encrypted data entirely, righ…

No, it's the other way around; a block overlap is effectively a guarantee it's the same file with the same key. Suppose they are using 1MB chunks. The keys are 16 bytes. The 1MB chunk may be 2 ^ (8 millionish) different things, the keys only 2^128 different things. Thus, two different 1MB chunks, to be the same file with different keys, must have two of their possible 2^128[1] encryptions overlap out of the possibili…

Why would you choose so large a block size for de-duplication? Your disk space savings decrease as block size increases. Choosing a more effective block size will increase the chance of collisions. And collisions are rather the desired goal of de-duplication.

Re: Megabad: A quick look at the state of Mega’s encryption

#52

Post by krallin »

el_cuadrado wrote: post #38
krallin wrote: post #14

The first part of the article is of unusually low quality for Ars Technica. It seems like the author is trying to bash Mega for being reportedly lazy in their implementation. This really is a shame because the parts regarding entropy, deduplication, and the conclusion, are very relevant! > Files and folders, therefore, are encrypted with symmetric encryption. Symmetric encryption means the same key is used to encrypt…

> Unless the article means that the key itself is a derivation (a hash?) of the password Why is this an issue? Other people call this 'issue' PKCS5.

In this specific case it wouldn't be very practical as changing the password would mean changing the encryption key to all your files. Hence, rendering all files unreadable.

That's why I suggested that a more appropriate application would be to derive a key from the password (using a PBKDF, which PKCS5 is (PBKDF2)) and use that one to encrypt the file encryption key.

This way, you can always change the password (provided you still have it) and encrypt the encryption key again, while not needing to encrypt each file.

Re: Megabad: A quick look at the state of Mega’s encryption

#53

Post by jerf »

roc wrote: post #51
jerf wrote: post #37

Earlier quoted context omitted.

No, it's the other way around; a block overlap is effectively a guarantee it's the same file with the same key. Suppose they are using 1MB chunks. The keys are 16 bytes. The 1MB chunk may be 2 ^ (8 millionish) different things, the keys only 2^128 different things. Thus, two different 1MB chunks, to be the same file with different keys, must have two of their possible 2^128[1] encryptions overlap out of the possibili…

Why would you choose so large a block size for de-duplication? Your disk space savings decrease as block size increases. Choosing a more effective block size will increase the chance of collisions. And collisions are rather the desired goal of de-duplication.

And all of your other costs associated with the block go up. While I do not work on the backup product my company produces, I've used it as a service. They've told me 1MB is pretty much too small nowadays, though it made sense when they started.

People have this weird fetish around deduplication, but it isn't magic. It makes it so the tenth copy of storing a backup of Windows XP doesn't hardly cost you any space. This is where the astonishing compression claims come from. It does not magically compress much of anything else, though. The claims are true, but not generally applicable. In practice, the middle bits of otherwise unrelated files don't get de-duped, excepting a couple of obvious and rare cases like 'a megabyte of zero padding in the middle of a file' which hardly amount to anything. Your World of Warcraft texture file is simply not going to overlap with your eBook copy of 50 Shades of Grey, and in general, two things sampled even from a 2^(524,288) possibility space, which corresponds to an absurdly small 64KB block size, are very unlikely to collide.

Block size has very little effect on collisions; what collides are identical files, or files that are nearly identical because they are versions of the same thing (and block size tends to matter surprisingly little there for various reasons), and what doesn't collide is everything else.

Re: Megabad: A quick look at the state of Mega’s encryption

#54

Post by roc »

jerf wrote: post #53
roc wrote: post #51

Earlier quoted context omitted.

Why would you choose so large a block size for de-duplication? Your disk space savings decrease as block size increases. Choosing a more effective block size will increase the chance of collisions. And collisions are rather the desired goal of de-duplication.

And all of your other costs associated with the block go up. While I do not work on the backup product my company produces, I've used it as a service. They've told me 1MB is pretty much too small nowadays, though it made sense when they started. People have this weird fetish around deduplication, but it isn't magic. It makes it so the tenth copy of storing a backup of Windows XP doesn't hardly cost you any space. Thi…

I'll readily concede that de-duplication isn't going to get you much, once you start talking about uniquely-encrypted data.

1MB just sounded much larger than what I've read and seen. At that size, yeah, about the only thing you're going to be de-duplicating is entire files. Interesting that smaller block sizes don't de-duplicate enough in practice to justify it.

Re: Megabad: A quick look at the state of Mega’s encryption

#55

Post by thirsteh »

B-Con wrote: post #17

> Unfortunately for Mega, it's generating the RSA keys with Javascript, and the method employed doesn't do a very good job at all of capturing entropy. I think that the vast majority of security experts advise that none of the crypto should be done in JavaScript, especially key generation. But I'm guessing they wanted to do it that way so that the key generation is done on the client side. They could generate high-qu…

Indeed. This is the Web Crypto API. It includes access to a CSPRNG: http://www.w3.org/TR/WebCryptoAPI/

Re: Megabad: A quick look at the state of Mega’s encryption

#56

Post by thirsteh »

jiggy2011 wrote: post #45

"Symmetric encryption means the same key is used to encrypt and decrypt your data; this is less secure than asymmetric encryption" Wat? If this is true, why does an RSA key have to be significantly longer than an AES key? In fact in most implementations asymmetric crypto is not used for exchanging or storing data. It's used to exchange a key that can be used for symmetric crypto.

It's not true. And yes, for equivalent bits of security, you need much larger keys with the RSA algorithm. The main reason it's not used for much beyond key-exchange is that it's thus exponentially slower.

Re: Megabad: A quick look at the state of Mega’s encryption

#57

Post by KMag »

jerf wrote: post #53
roc wrote: post #51

Earlier quoted context omitted.

Why would you choose so large a block size for de-duplication? Your disk space savings decrease as block size increases. Choosing a more effective block size will increase the chance of collisions. And collisions are rather the desired goal of de-duplication.

And all of your other costs associated with the block go up. While I do not work on the backup product my company produces, I've used it as a service. They've told me 1MB is pretty much too small nowadays, though it made sense when they started. People have this weird fetish around deduplication, but it isn't magic. It makes it so the tenth copy of storing a backup of Windows XP doesn't hardly cost you any space. Thi…

On a related note, if any of you are designing a file format, please place metadata at the end, so multiple copies of the same file with different metadata can be easily deduplicated. If the metadata is variable length and at the beginning, the block boundaries get shifted. In order to deal with shifted files, one ends up using a pseudorandom function of block contents to decide where the dedup block boundaries are (rather than some multiple of the filesystem block) in order to be able to re-sync the dedup block boundaries between the two files beginning with different length metadata.

For that matter, if you're writing a library to handle an existing file format that has a flexible location for metadata, please put any metadata (especially user-generated metadata) as far back in the file as possible.

Of course, the main advantage is that you don't have to shift all of the data around if the user starts editing file metadata, but it does also help block-level deduplication.

Re: Megabad: A quick look at the state of Mega’s encryption

#58

Post by summerdown2 »

simias wrote: post #4

I don't understand that part: Symmetric encryption means the same key is used to encrypt and decrypt your data; this is less secure than asymmetric encryption (where one key encrypts and a different key decrypts), but it's faster and easier to implement. Isn't that comparing apples to oranges? What would be the benefit for Mega or the user to switch to RSA for that? Encryption would be using a symmetric cipher anyway…

It's now been changed to:

> Symmetric encryption means the same key is used to encrypt and decrypt your data; this less computationally-intensive and easier to implement than asymmetric encryption, which we'll get to in a moment.

Re: Megabad: A quick look at the state of Mega’s encryption

#59

Post by loeg »

B-Con wrote: post #31
loeg wrote: post #22

Earlier quoted context omitted.

But both clients still need to be able to decrypt it with their separate keys... even if the cleartext hash matches, the stored data is ciphertext and can only be encrypted with one user's key.

It's possible: http://crypto.stackexchange.com/questions/729/is-convergent-... . The trick lies in ensuring the key used for encryption is linked to the file, not the user. You then have to store a separate key for each encrypted file (and that encryption key is probably going to need to be wrapped by your actual user key for storage).

True.

Re: Megabad: A quick look at the state of Mega’s encryption

#60

Post by loeg »

B-Con wrote: post #31
loeg wrote: post #22

Earlier quoted context omitted.

But both clients still need to be able to decrypt it with their separate keys... even if the cleartext hash matches, the stored data is ciphertext and can only be encrypted with one user's key.

It's possible: http://crypto.stackexchange.com/questions/729/is-convergent-... . The trick lies in ensuring the key used for encryption is linked to the file, not the user. You then have to store a separate key for each encrypted file (and that encryption key is probably going to need to be wrapped by your actual user key for storage).

But, how does the second user to upload the same file know the file's encryption key (since presumably the server only knows ENCRYPT(FILE_KEY with USER1_KEY))?

It seems like either the server must know the file's encryption key, or the key must be derivable from the contents... which is no good.

Post reply on HN