Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Should you encrypt or compress first?”

blog.appcanary.com

Post reply on HN
Print view View original

101–110 of 249 posts

Re: “Should you encrypt or compress first?”

#101

Post by lisper »

belorn wrote: post #80
lisper wrote: post #39

Earlier quoted context omitted.

> If security is important, the answer to that is no It's a little more nuanced than that. Compression may cause information leaks or it can prevent them depending on the circumstances. If you're encrypting an audio stream, then compressing it first can cause leaks. If you're encrypting a document, then compressing it first may prevent leaks.

Could you elaborate what kind of scenario would work on a audio stream but not on a document? In order to use CRIME, the attacker would need to repeatably injected content before the compression stage in order to test if it matches some part of the source. Audio is kind of heavy uncompressed and aren't normally streamed as such.

A raw audio stream is a constant bit rate stream. If you compress it, it becomes (in general) a variable bit rate stream. The bit rate contains information about the raw audio which survives encryption.

If you compress a document that simply obscures the size of the original document.

Re: “Should you encrypt or compress first?”

#102

Post by Miner49er »

richard_todd wrote: post #72
danielweber wrote: post #21

Earlier quoted context omitted.

"Don't compress at all" is the better default answer. Noting that "compression after encryption is stupid."

If security is the top concern, making all encrypted messages the same length would be ideal as far as I can tell. That way, all you are giving away is an upper bound on the message size. Padding with random noise to a uniform length (with the payload either compressed or not) and then encrypting should be the most secure option.

What's the point of compressing if you're just going to pad it anyway? If you're worried about security just encrypt and don't compress.

Re: “Should you encrypt or compress first?”

#103

Post by api »

TorKlingberg wrote: post #89
api wrote: post #52

Earlier quoted context omitted.

The rule is: never compress something secret together with something potentially attacker-influenced. If the attacker can influence the traffic, they can potentially gather information about the secret by examining the effect of differing traffic patterns on the size of the encrypted result.

No, the rule is: never compress something secret together with something potentially attacker-influenced, at least if the length of the compressed data leaks.

Fixed. That was a typo/thinko.

Re: “Should you encrypt or compress first?”

#104

Post by [deleted] »

TorKlingberg wrote: post #89
api wrote: post #52

Earlier quoted context omitted.

The rule is: never compress something secret together with something potentially attacker-influenced. If the attacker can influence the traffic, they can potentially gather information about the secret by examining the effect of differing traffic patterns on the size of the encrypted result.

No, the rule is: never compress something secret together with something potentially attacker-influenced, at least if the length of the compressed data leaks.

[deleted]

Re: “Should you encrypt or compress first?”

#105

Post by AlyssaRowan »

belorn wrote: post #80
lisper wrote: post #39

Earlier quoted context omitted.

> If security is important, the answer to that is no It's a little more nuanced than that. Compression may cause information leaks or it can prevent them depending on the circumstances. If you're encrypting an audio stream, then compressing it first can cause leaks. If you're encrypting a document, then compressing it first may prevent leaks.

Could you elaborate what kind of scenario would work on a audio stream but not on a document? In order to use CRIME, the attacker would need to repeatably injected content before the compression stage in order to test if it matches some part of the source. Audio is kind of heavy uncompressed and aren't normally streamed as such.

Imagine a voice call, using a variable-bitrate codec, broken into small packets (not necessarily over the internet), of interest to a passive eavesdropper who cannot decrypt the content, but can observe metadata: the timing of packets and their sizes.

Feed that information into a prebuilt phoneme-in-context model trained on that codec and language, and said eavesdropper could probably reconstruct a pretty good estimated transcript - different phonemes compress differently.

There is no reason this would not work in bulk: I am under the impression GCHQ have done a fair amount of research in this area, for example.

Re: “Should you encrypt or compress first?”

#106

Post by zardeh »

DanBlake wrote: post #90
IanCal wrote: post #58

Earlier quoted context omitted.

> It wont be a huge difference in file size, but it will be smaller. may be smaller, not all outputs will be smaller, this is true of all lossless compression algorithms: https://en.wikipedia.org/wiki/Pigeonhole_principle

This is literally all I was saying and people went on a downvote frenzy. People are literally saying that a encrypted output can NEVER be compressed. I was saying that it can (although not all the time!) and the gains would be minimal. Ive corrected the first post with the word 'may'

But, (also by the same ideas), most encrypted texts will not be able to be compressed by more than a single bit.

Re: “Should you encrypt or compress first?”

#107

Post by olliej »

DanBlake wrote: post #91
zeveb wrote: post #87

Earlier quoted context omitted.

> A encrypted file can very well have something like 100 X's in a row Any decent encryption algorithm has output indistinguishable from random noise; the odds of 100 Xes in a row in random data is 1:2^800; there are only 10^80 particles in the universe (2^266), which is 534 base-2 orders of magnitude smaller than 2^800. Yes, every particle in the universe could be a universe of fundamental particles and you'd still n…

Feel free to upload a 5MB file under your narrow restrictions and I will see if I can reduce it with one of the 20 different compression systems available to me.

If there is structure in an encrypted file then the encryption is broken. It is mathematically indistinguishable from true random data. No compression algorithm can compress it.

An additional point you need to consider: You can select different models and compression programs (and some programs do do this), but you have to record which one you use in your output. This again increases the size of the output - this means that even a no-op compression algorithm (say cat or cp :D ) results in output that is bigger (you need a initial byte to say you haven't tried to compress).

I think the better approach to what you're claiming is for you to provide an example that is compressed. Just provide the input, the key, the encryption algorithm (AES-GCM for preference, but seriously AES-CBC would work to), and the compression algorithm.

Re: “Should you encrypt or compress first?”

#108

Post by kaoD »

hinkley wrote: post #65
vog wrote: post #26

Earlier quoted context omitted.

This is exactly what the article says.

Does this really need to be said though? I may be too close to the problem. I've had to explain this to project managers and customers of course, but this is hacker news. It feels like a three page article on why you should put your socks on before your shoes and not after.

> but this is hacker news.

So? There are too many domains to be familiar with. You just happen to be familiar with this one. I bet there are plenty things I'd consider obvious on many other domains where you'd be clueless.

Re: “Should you encrypt or compress first?”

#109

Post by olliej »

DanBlake wrote: post #91
zeveb wrote: post #87

Earlier quoted context omitted.

> A encrypted file can very well have something like 100 X's in a row Any decent encryption algorithm has output indistinguishable from random noise; the odds of 100 Xes in a row in random data is 1:2^800; there are only 10^80 particles in the universe (2^266), which is 534 base-2 orders of magnitude smaller than 2^800. Yes, every particle in the universe could be a universe of fundamental particles and you'd still n…

Feel free to upload a 5MB file under your narrow restrictions and I will see if I can reduce it with one of the 20 different compression systems available to me.

Ooh, the encryption comment I just made reminded me of an example of compression potentially working on encrypted content.

AES-ECB mode will actually compress fairly well for some inputs (the canonical example being a bitmap image). The reason the compression can work? Because the crypto is broken. In the image case you can actually just visually (no maths or anything) see a large amount of the details of the source image.

Re: “Should you encrypt or compress first?”

#110

Post by kbenson »

AlyssaRowan wrote: post #105
belorn wrote: post #80

Earlier quoted context omitted.

Could you elaborate what kind of scenario would work on a audio stream but not on a document? In order to use CRIME, the attacker would need to repeatably injected content before the compression stage in order to test if it matches some part of the source. Audio is kind of heavy uncompressed and aren't normally streamed as such.

Imagine a voice call, using a variable-bitrate codec, broken into small packets (not necessarily over the internet), of interest to a passive eavesdropper who cannot decrypt the content, but can observe metadata: the timing of packets and their sizes. Feed that information into a prebuilt phoneme-in-context model trained on that codec and language, and said eavesdropper could probably reconstruct a pretty good estima…

So, it's not just compression per se, but compression plus real-time plus some knowledge about the underlying information.

These sorts of attacks don't matter against a compressed and then encrypted file of indeterminate data, correct? Similarly, an amalgamation of many types of data, such as a disk or archive?

Post reply on HN