The story around the Linode hack • Hacker News Board

Re: The story around the Linode hack

#131

Post by eridius » Wed, May 08, 2013, 1:43 AM UTC

marshray wrote: post #129

eridius wrote: post #122

Earlier quoted context omitted.

The encryption keys are useless. The decryption keys are what's important. You do have a point that they could have theoretically intercepted HTTP(S) POSTs, but I don't think anyone's claimed that they actually did that.

They were using asymmetric crypto on their CC data column?

Yes. Linode publicly stated that they were using public-key cryptography, that the private key was secured with some crazy-long passphrase, and that the passphrase wasn't stored digitally, meaning that once a month when they billed they had to manually enter the private key.

So for the hackers to get the decrypted private key, then either Linode must have royally screwed up and kept the decrypted key in-memory during the rest of the month (which seems rather unlikely), or the hackers must have had control of the machine during the time in which they did billing (which I don't think is true, because billing presumably happens either at the start or the end of the month, and didn't the hack take place a bit earlier than that?).

So yeah, I believe them when they said they got the private key. But nobody's said anything to convince me that they got the _decrypted_ private key. And if the passphrase really is as long and complex as Linode claims, then it should be reasonably secure (caveat: I am not a security researcher, or otherwise qualified to judge the security of anything).

Re: The story around the Linode hack

#132

Post by marshray » Wed, May 08, 2013, 1:44 AM UTC

eridius wrote: post #131

marshray wrote: post #129

Earlier quoted context omitted.

They were using asymmetric crypto on their CC data column?

Yes. Linode publicly stated that they were using public-key cryptography, that the private key was secured with some crazy-long passphrase, and that the passphrase wasn't stored digitally, meaning that once a month when they billed they had to manually enter the private key. So for the hackers to get the decrypted private key, then either Linode must have royally screwed up and kept the decrypted key in-memory during…

OK, I remember reading that now.

Good move on their part.

Re: The story around the Linode hack

#133

Post by tiredofcareer » Wed, May 08, 2013, 1:58 AM UTC

lelandbatey wrote: post #126

threeseed wrote: post #123

Earlier quoted context omitted.

I have found lots of the smaller VPS providers (especially those that provided dedicated/colo services) to have great service. Check on http://webhostingtalk.com I fail to understand how you can think Linode has a great track record. It is an unequivocal disgrace. TWICE they have mislead their own customers over a major security incident. And there have been lots of times during outages (in particular my time at Frem…

I give Linode a lot of slack. When people say "Oh, they should be more secure" I often say "Really?" In an ideal world, yes, they should be more secure. However, as in this case, they got taken advantage of via a zero-day attack, with others planned well outside the scope of what Linode could have planned for. Which is insane. Can you even name something, anything that they could have done to protect themselves? Addi…

While what you say has merit, Linode's actions demonstrate an ambivalence toward security. Public key encryption for card numbers (yay!). The private key stored on the same machine and the key loaded in memory (boo!). ColdFusion was not properly secured (simply preventing access to /CFIDE would have neutralized this vector) and they focused first on preserving themselves. I've also been personally annoyed when there are sweeping outages and information is withheld for seemingly arbitrary reasons. Support is apparently instructed to be vague. It's overwhelmingly frustrating.

I'm just as annoyed at Amazon for this, to be honest, and in the large, annoyed at our industry for being so unnecessarily secretive. We need to stop thinking of our infrastructure as our competitive advantage; to pick on Google as an example, while Google are obviously masters of running systems at scale, their infrastructure efficiency is not the reason people choose Gmail. Obviously their platform gives them some competitive advantage but, for example, their policy of withholding even the innocuous names of internal systems is bizarre. I think the rest of the industry follows that lead.

It's weird that we embrace openness in the FLOSS communities but when it's time to build a revenue-making company, the details of the inner workings are immediately a hush-hush secret. If you're doing something simple enough that describing it means someone can replicate it, it's an idea that can be replicated trivially anyway. I bet everybody in hosting knows how Linode works, and I doubt there's any kind of espionage taking place.

In this case, it's fine to be secretive if you'd like, but at least tell me how you plan to prevent the problem from recurring. Linode always says "we're working diligently to prevent this from happening again" but provides no details whatever. The announcement from the founder of Linode[1] underlines this; the entire tone of the post is "here's how we band-aided the immediate problem," with no details on where they go from here as a business or culture.

Re: The story around the Linode hack

#134

Post by jrochkind1 » Wed, May 08, 2013, 2:01 AM UTC

jrockway wrote: post #130

crisnoble wrote: post #110

Earlier quoted context omitted.

Everyone bitching about HTP or AnonOps or any other hacking group that likes bragging should at least be thankful that they talk about their hacks. I would bet that the crime syndicates have better hacks and keep their mouths shut about them. Those vulnerabilities don't get patched, those customers never get notified. I am not defending HTP or the like, just saying, at least they boast.

I worry more about governments than organized crime these days.

Yep, them too, don't tell anyone about their hacks.

(I mean, "More about governments than organized crime? There's a difference?")

Re: The story around the Linode hack

#135

Post by tiredofcareer » Wed, May 08, 2013, 2:11 AM UTC

marshray wrote: post #120

tiredofcareer wrote: post #12

Some hopefully-helpful clarifications of the inside baseball talk from just the overview (I haven't read the full zine), enhanced with inside and general knowledge I've gained in my travels on this mortal coil: - HTP claims to have{, had} access to name.com, which Linode currently uses. This access enables an unauthorized party to update authoritative nameservers for your domain; i.e., if you host at Amazon, very lik…

> The access that HTP obtained does not, full stop, lead to root on Linode instances without at least one shutdown job or change of root password job showing up in your Linode's history that you did not ask for. ... the access they obtained does not lead to root on the Linode host fleet itself I wouldn't bet on that. > There will always be targets but harboring SwiftIRC is probably a malicious-actor magnet. Isn't tha…

> I wouldn't bet on that.

I don't need to wager, and can speak with authority based on what I know (which I'd prefer to leave vague). There are two vectors into a Linode's filesystem from the perspective of an internal attacker: having root on the Xen host or gaining a login on the Linode. Knocking over the database and Web server gives you neither unless the person reused their account's password as their root password, in which case it's behind a cryptographic hash and subject to the typical rules there. If you own the database, you do have LISH access which gives you the equivalent of a VGA console; if someone left that console logged in, it's a vector as well.

The only vector HTP would have had in the general case would be bouncing the Linode. It's a fairly sufficient air gap, in a way.

Re: The story around the Linode hack

#136

Post by jrochkind1 » Wed, May 08, 2013, 2:12 AM UTC

tiredofcareer wrote: post #85

VLM wrote: post #41

Earlier quoted context omitted.

"because you know, that's a good target to burn registrar access on and all" No, nearly ideal use. Its like a strategic nuclear weapon. You don't use it sneakily, that's the opposite of the whole point. Always intimidate as publicly as possible and in the tech community messing with linode is about as public as it gets. The other part is showing off, its like declaring "we have access to them all but we don't care ab…

They didn't utilize the access to go after Linode. They intended to utilize it to go after SwiftIRC, which nobody gives a shit about. That's where my comments came from. Linode just happened to be a nice prize on the way.

Yeah, I mean, it's supposed to come off as showing off, right? "Sure, we're so crazy good, that we can take out name.com and linode just cause some script kiddies pissed us off, no sweat. Don't mess with us. And we're so in it for the lulz, that' SURE we'd take out linode and name.com just to get some script kiddies, and not bother trying to sell the CCs or anything."

Or, they're lying. I mean, they could be lying about the whole damn story, who knows (not me).

Re: The story around the Linode hack

#137

Post by CleanedStar » Wed, May 08, 2013, 2:13 AM UTC

davidw wrote: post #15

RoboTeddy wrote: post #13

Here's an attempt at an explanation/translation: HTP ("Hack The Planet") is a group that likes to break into things. Another (unnamed) group of people impersonated a third group of people ("ac1db1tch3z") and tried to cause trouble for HTP. The impersonators located HTP by examining one of HTP's botnets (a collection of compromised computers that are used to launch things like denial of service attacks). Botnets have…

> tried to cause trouble for HTP. Here's hoping the FBI "causes trouble" for the lot of them. Breaking into other people's stuff is not cool. If I leave my door open by mistake, yes, that makes me a bit absent minded, or foolish, but it does not give anyone the right to wander into my house.

I'm afraid your analogy is stretched a little too far for me - that your open door leading to someone physically wandering into your personal living space is the same as some corporation with tens of millions of revenue a year that has some script kiddie sitting in his mom's basement seeing some Linode stuff come onto his screen. I guess when I think of some kid snooping around some corporation's computers, I'm supposed to compose a mental image of some thug physically invading my own home. Yes, I can see why corporations want people to think this way, but it's a rather silly metaphor as far as I'm concerned.

Re: The story around the Linode hack

#138

Post by jrochkind1 » Wed, May 08, 2013, 2:15 AM UTC

tiredofcareer wrote: post #37

sillysaurus wrote: post #35

Earlier quoted context omitted.

Adding a conditional ("do I answer or do I proxy?") on every DNS query -- and there are many -- is going to introduce enough latency to be noticed unless you throw a lot of gear at it. Hm, why? Any modern CPU is blazingly fast. Writing it in Ruby probably wouldn't be smart, but Python + PyPI or Lua + LuaJIT would easily get within a factor of 10x of C.

I didn't say it would be technically impossible, I said it would be noticed. If you make it a theoretical problem, and it most certainly isn't (there are a lot more practicalities involved), you're adding at least another string compare to every query. That's enough of a latency shift for me to notice in my graphs -- I notice when the Internet reroutes itself and my DNS latency goes up by 5 milliseconds. This isn't a…

Okay, so, when you notice your DNS latency going up by 5ms... how much investigation do you then do to confirm exactly what caused this, and have a very high confidence (how high?) of ruling out it being caused by a MitM on the DNS? Really?

Re: The story around the Linode hack

#139

Post by tiredofcareer » Wed, May 08, 2013, 2:17 AM UTC

eridius wrote: post #131

marshray wrote: post #129

Earlier quoted context omitted.

They were using asymmetric crypto on their CC data column?

Yes. Linode publicly stated that they were using public-key cryptography, that the private key was secured with some crazy-long passphrase, and that the passphrase wasn't stored digitally, meaning that once a month when they billed they had to manually enter the private key. So for the hackers to get the decrypted private key, then either Linode must have royally screwed up and kept the decrypted key in-memory during…

> So for the hackers to get the decrypted private key, then either Linode must have royally screwed up and kept the decrypted key in-memory during the rest of the month (which seems rather unlikely),

They bill you the moment you add a Linode, automatically, if your credit is not sufficient to cover the new Linode. Careful walking that assumption too far; I think it's safe to say the key was kept in memory.

Re: The story around the Linode hack

#140

Post by tiredofcareer » Wed, May 08, 2013, 2:22 AM UTC

white_devil wrote: post #114

tiredofcareer wrote: post #12

Some hopefully-helpful clarifications of the inside baseball talk from just the overview (I haven't read the full zine), enhanced with inside and general knowledge I've gained in my travels on this mortal coil: - HTP claims to have{, had} access to name.com, which Linode currently uses. This access enables an unauthorized party to update authoritative nameservers for your domain; i.e., if you host at Amazon, very lik…

It bugs me that apparently, everything I'll ever host can just be "owned" at will by some random bunch of hackers doing whatever they feel like doing. Is it feasible for a "mere mortal" to stay safe?

User input is trying to kill you until proven otherwise. And even then, it probably still is. If you accept anything from a user, treat it like you're carrying spent nuclear fuel around your application; that doesn't just mean form inputs, either; sometimes you have to wear gloves that even consider files read from the filesystem as suspicious. There are numerous attacks on temporary files if implemented incorrectly.

Damned near every exploitable security problem in an application can be traced back to this one simple rule. XSS is a common one. The ones that don't conform to the rule are rare and magical, like unicorns, and usually involve something exotic like hardware side channels. If you take input it will be abused, so plan accordingly. That occasionally manifests in unexpected ways like timing attacks, in which case an attacker repeatedly sends you lots of carefully-chosen input to deduce something based on how long it takes your code to reply, much like cracking a safe. It's a basic attack on a string compare, which is O(n). This simple code is vulnerable:

    def price_multiplier(form):
        if form.input == 'super-sekrit-coupon-code':
            print >>browser, "100% discount applied!"
            return Decimal('0.0')
        print >>browser, "Sorry, that's an invalid coupon!"
        return Decimal('1.0')

A timing attack would try to deduce each letter here, since the string comparison will short-circuit once it finds a wrong character. Using a simplistic model, say I sent 'a' through 'z'; 's' took 10 time units to respond, but the other characters took 8. Now I try 'sa' through 'sz', and go from there. As a thought exercise, try to imagine a few ways to prevent it, and consider the pros and cons of each; now you're thinking in the security mindset.

Securing an infrastructure is another story, but the rule is applicable with the occasional clarification to everything there, too. There are domains of trust even within your own organization; malicious employees will try to harm you, as well, and you get little to no warning of those. Do your interns need root on machines containing customer data? Employees are users too.