The part I still don't get because I haven't seen it discussed so far is that their original patch mentionned they had been using a variant of the patch "for a decade" but I never got why they didn't seek to get it merged upstream a decade ago. Or why they released "a variant" of their patch instead of the real thing to begin with.
Heartbleed Update v3
11–20 of 55 posts
Re: Heartbleed Update v3
#12Good politics: they had custom code for parts of OpenSSL, they published it when it seemed needed, open-source community told them what was wrong, they fixed it and apologized. No bullshit.
Also: the way this played out makes an awesome story for the next time Akamai experiences internal pushback against participating in open source. "Remember that one time it saved our bacon" is a political goldmine.
They are only in problem now, because they have released the patch. If no one outside the company saw that patch, they could spot the problem and fix it without anyone noticing and creating a PR disaster.
They probably wouldn't, but they could.
That is one argument someone could use against "participating in open source".
It's a bit to late for saving bacon, patch should be publicly reviewed before it went into production. So no bacon saved, but there is a hope for a smoke detector.
Re: Heartbleed Update v3
#13What's interesting is that I just watched a video that strongly relates to this problem last night (from DJB at CCC):
https://www.youtube.com/watch?v=IuSnY_O8DqQ
It is only part of the video, but they talk about searching Google and finding partial private keys, e.g. ---- BEGIN PRIVATE KEY ---.
Even with text missing, it is trivial to recover the entire key, because of redundancy that Akamai neglected to take into account. I never really thought about how those ASCII strings are encoded, but it's actually the 6 or so values encoded. If some are missing, you can recover the others.
In retrospect, why shouldn't the private keyh have RFC 822-like key value format so this is obvious? I don't see why it has to be encoded at all; the format is unnecessarily obscure.
The rest of this talk is also great. Most of it is accessible to non-cryptographers (like me), as long as you understand basic arithmetic and have an undergrad-level overview of public key crypto (which they give). They provide very readable and runnable Sage scripts.
I really like the batch GCD attacks. Very elegant and effective.
Re: Heartbleed Update v3
#14I would like to know how many security related patches they are keeping for themselves? And I think not only Akamai did this. I hope that this situation will prove that security needs a bit more patch reviews and a bit less of secret sauce.
I suspect their patches are oriented toward paranoia. Their particular patch (keeping what they believed to be all critical values in a special region of memory used only for critical things) had no obvious necessity -- it didn't obviously provide more security -- but was just paranoia. Paranoia isn't always necessary, or even desirable; hence, the remainder of the patches have no obvious external utility, so they presumably haven't released them.
They may also have performance improvements, conceivably, and those have a more obvious rationale for keeping secret.
Re: Heartbleed Update v3
#15Willem's letter to Akamai addressed three areas where their patch was likely insufficient:
(1) First question is then, are 'p', 'q' and 'd' the only sensitive parameters? NO. (The Chinese Remainder Theorem parameters are included in OpenSSL private keys by default and they are enough to recover the private key).
(2) Second question: Does OpenSSL ever copy the private key, or parts, after the key has been read? YES.
(3) Third question: Does OpenSSL allocate temporary variables for intermediates when performing operations with a private key. Can they be used to recover the private key? YES and YES.
(Willem's post submitted to HN: https://news.ycombinator.com/item?id=7582990 )
This update from Akamai only addresses (1), AFAICT.
Re: Heartbleed Update v3
#16Earlier quoted context omitted.
Also: the way this played out makes an awesome story for the next time Akamai experiences internal pushback against participating in open source. "Remember that one time it saved our bacon" is a political goldmine.
They are only in problem now, because they have released the patch. If no one outside the company saw that patch, they could spot the problem and fix it without anyone noticing and creating a PR disaster. They probably wouldn't, but they could. That is one argument someone could use against "participating in open source". It's a bit to late for saving bacon, patch should be publicly reviewed before it went into produ…
If they had not released the patch, then it's entirely possible that black hats would have been able to make off with the keys from their customers - and that would have been a real disaster.
Re: Heartbleed Update v3
#17Willem's letter to Akamai addressed three areas where their patch was likely insufficient: (1) First question is then, are 'p', 'q' and 'd' the only sensitive parameters? NO. (The Chinese Remainder Theorem parameters are included in OpenSSL private keys by default and they are enough to recover the private key). (2) Second question: Does OpenSSL ever copy the private key, or parts, after the key has been read? YES. (…
Why would their update need to address more than just (1)? That's the only part of the problem that they have to deal with. The OpenSSL team should be the one answering (2) and (3), since it's OpenSSL that copies the PK/parts after it's been read, or allocate temp vars for intermediates. If Akamai chooses to patch those two problems and then releases it to open source like their initial patch, then they can answer them, but otherwise, what's the point?
Re: Heartbleed Update v3
#18Good response from Akamai. No beating around the bush there.
No, this was not a good response. It was well written and took responsibility for bad code, but it failed to address the elephant in the room: why were keys not immediately rotated?
I've said this a number of times lately, but it bears repeating that the golden rule of security is: If a compromise occurs, assume that everything is compromised. Even if their approach was tested to be perfect, they shouldn't have made the assumption that that actually reflected reality.
They were extraordinarily irresponsible. There's simply no other way to slice it.
Re: Heartbleed Update v3
#19Willem's letter to Akamai addressed three areas where their patch was likely insufficient: (1) First question is then, are 'p', 'q' and 'd' the only sensitive parameters? NO. (The Chinese Remainder Theorem parameters are included in OpenSSL private keys by default and they are enough to recover the private key). (2) Second question: Does OpenSSL ever copy the private key, or parts, after the key has been read? YES. (…
They also said
> In parallel, we are evaluating the other claims made by the researcher, to understand what actions we can take to improve our customer protection.
which seems to imply that they're at least looking into the other questions.
Re: Heartbleed Update v3
#20Why people are only talking about SSL keys? Because Heartbleed allowed to fetch any data, that could have also included SSH keys and or any other authentication credentials / information the server got, for itself or to other services. As example if your site got Facebook API keys it's possible that those have been leaking too.