Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Security incident update

blog.linode.com

Post reply on HN
Print view View original

271–280 of 282 posts

Re: Security incident update

#271

Post by jmillikin »

steveklabnik wrote: post #266
jmillikin wrote: post #251

Earlier quoted context omitted.

It's not a contradiction, because 1) the webserver would not have login credentials to the hardened machine and 2) the interface would be much smaller than "everything in SQL". A typical example of such an interface would expose about four operations: // Return true if the given user/password pair is valid. bool PasswordValid(string user, string password) // Return true if the password was changed successfully. bool…

That'd still be 'internet-facing' though, by your definition. Yes, it might require one more round of cracking (the simple interface), but the machines are still connected.

No, it would not. Please read my post again. The webserver would not have login credentials for the password management machine.

Additionally, I object to your implication that any network service can be trivially compromised with "one more round of cracking". This is a view of software security that is propagated by science-fiction movies, and is not grounded in real life. Vulnerabilities that permit remote code execution are relatively easy to prevent by taking appropriate precautions when designing and implementing a network service, and almost every such vulnerability has its root cause in doing something obviously insecure (e.g. writing in C/C++ or a language with eval(), passing user input into an OS procedure).

Re: Security incident update

#272

Post by corford »

nenolod wrote: post #270
corford wrote: post #268

Earlier quoted context omitted.

What exactly do you propose as an alternative? At some point the card details need to be decrypted. Security always boils down to somewhere between total lock down and being able to actually get things done. The only semi reasonable alternative would be not to store any CC details and outsource that task to a payment processing company. It's not a great alternative though as you'd effectively be giving up the ability…

Lets be honest here. How frequently do you think a company changes processors?

Agreed, however, imho you'd have to be insane to run a company with hundreds of thousands of monthly billed customers and turn control of the card details over to a third party (who, lets be honest, will be running their backend in a very similar manner i.e. encrypted CC details protected with a passphrase). I'd take my chances the same way Linode chose to rather than punt the issue upstream and hope for the best.

I might be biased though as my default approach to most things is to do (and have control over) as much as possible yourself unless there are extremely good, unequivocal reasons not to.

Re: Security incident update

#273

Post by thaumaturgy »

corford wrote: post #267
lawnchair_larry wrote: post #213

Earlier quoted context omitted.

I understand this, and any system that needed this information would not work if it relied on a human to decrypt.

I think the most likely answer is pretty simple and non shocking: the key is loaded via an agent on boot (subject to the correct passphrase being entered) and stays resident in memory (either permanently or for x minutes/hours). Where's the problem?

If they did it that way, then because the attackers had access to the keys and the server code for an unknown period of time (up to maybe a couple of days), they may have been able to either extract it from memory or use the private key with someone else's credentials to decrypt some or all of the payment information.

I'm really trying to avoid speculation at this point -- we simply don't have enough information to make smart guesses -- but I think we can reasonably conclude that nobody at Linode was manually keying in a password every time there was a charge to a customer card, and therefore it's not insanely paranoid to cancel any cards associated with Linode.

Really the most annoying thing here is Linode's obtuseness about the attack and their processes and what's at risk, and if I end up moving to another service (as at the moment I'm inclined to), that will be the reason -- not because they got compromised, but because they valued their own butts over their customers'.

They're a VPS service for chrissake. They could explain in very technical terms exactly how manager passwords were stored and how the credit card information was stored & accessed and what other billing information may have been compromised -- a full disclosure -- and most of their customers would not only understand most of it, but they would appreciate it.

Sorry, I kinda went off on a tangent there, but the more I think about how this has been handled, the less I feel like I can trust them.

Re: Security incident update

#274

Post by thaumaturgy »

dangrossman wrote: post #222
jmillikin wrote: post #221

Earlier quoted context omitted.

The attacker claimed that he compromised their webserver. If he had access to the private key and encrypted numbers, then they were indeed being stored on an internet-facing machine. 07:52 the CCrypter class of the linode application context was accessable from outside the wwwroot using undocumented ColdFusion methods. i was fully able to decrypt the ccs using the in-memory privkey that they supplied the password for…

Let's make this more clear: Linode has two servers. One is a web server that runs their website and contains its source code. One is a database server containing customer information and encrypted credit card numbers. The web server is connected to the internet. The database server is not. They communicate over the private network. The web server is compromised. The attacker has the source code, which includes the IP…

> There's no way to build this that doesn't have a path from the internet to the card numbers either, since eventually they need to get onto an internet-connected machine to talk to the payment gateway and actually charge those cards.

I have to disagree.

Web server has write-only access to a remote database table on the accounts/billing server, writes account change requests; accounts/billing server polls table, matches account numbers to billing information, submits charge requests to payment gateway via a firewall that allows access to only a single IP or set of IPs; accounts/billing server writes charge result back to another table; web server has read-only access to the results table and subsequently changes the user's account type or sends out an email or performs other administrative duties.

That's how I'd build it, I've done similar stuff before. (I have a paranoid business client.)

The nice thing about this is that you can capture credit card information on the web server, encrypt it with the public key, and then decrypt it using the private key which is only on the accounts/billing server.

Getting unauthorized remote access to the accounts/billing server would be a challenge. If they needed authorized remote access to it for customer service reasons on ports other than ssh, I'd use authpf on the pf firewall sitting between the accounts/billing server and the rest of the network, along with a unique password-protected ssh key for each employee. But, really, 1 or 2 sysadmins are the only people who should have non-console access to it.

Re: Security incident update

#275

Post by prawn »

hornd wrote: post #175
mekoka wrote: post #85

The one thing that puzzles me above anything else about this whole thing is: "Linode uses ColdFusion". I mean, you have this Linux company that's pretty much an open source champion to us Linux guys and then... Adobe ColdFusion. Really? I know nothing about ColdFusion, don't get me wrong, I just find it an incredibly odd choice for a Linux company. That having been said, I think Linode just learnt a really unpleasant…

Perhaps they simply enjoy using ColdFusion. It gets a lot of hate, but some of the features that ship with CF (templating, extremely easy db access, PDF generation) made/make CF a rather powerful and fun language. If only Adobe had chosen to release it for free... You're right about it being an odd choice for a Linux company, though.

You can build on CF for free using Railo.

Re: Security incident update

#276

Post by stock_toaster »

eridius wrote: post #263
thaumaturgy wrote: post #260

Earlier quoted context omitted.

I mentioned this elsethread, but I've had changes to my account get charged to my card immediately (within minutes), even in the middle of the night. Last example of this was January 2012, so they may've changed their billing practices since then.

I suppose they could also keep the decrypted key in memory using something like gpg-agent.

I read somewhere (can't remember where now) that the hacker supposedly dumped the key from memory. My guess is they type in the passphrase on boot or something, and it lives in an agent thereafter.

Re: Security incident update

#277

Post by rjbond3rd »

laumars wrote: post #147
rjbond3rd wrote: post #146

Earlier quoted context omitted.

I disagree. First, it's hard to find good developers, period. S econd, in general, good developers are hard to find by design (e.g., to avoid recruiters, but also because they are most tof the time already doing interesting work). Lastly, yes, it's hard to find good Perl programmers to do boring work, under boring conditions, at a boring wage. But then that's true of all good developers. Try to find a Rails programme…

You're missing my point, PHP developers that know how to build websites are available ten to the dozen. But Perl most of the Perl developers we've tried to recruit over the last 18 months are either completely clueless at writing RESTful APIs or just terrible developers (sometimes even both). We've had far more success at hiring good PHP developers and teaching them Perl then we have had at finding the Perl developer…

Very cool! Build vs. buy, you're building, everyone wins (plus you're a great employer for training your devs).

What's your product or service? Now Perl programmers are now seeking -you- out :)

And I'm sure you already know about the YAPC job fair: http://www.yapcna.org/yn2013/job-fair.html

Re: Security incident update

#278

Post by laumars »

rjbond3rd wrote: post #277
laumars wrote: post #147

Earlier quoted context omitted.

You're missing my point, PHP developers that know how to build websites are available ten to the dozen. But Perl most of the Perl developers we've tried to recruit over the last 18 months are either completely clueless at writing RESTful APIs or just terrible developers (sometimes even both). We've had far more success at hiring good PHP developers and teaching them Perl then we have had at finding the Perl developer…

Very cool! Build vs. buy, you're building, everyone wins (plus you're a great employer for training your devs). What's your product or service? Now Perl programmers are now seeking -you- out :) And I'm sure you already know about the YAPC job fair: http://www.yapcna.org/yn2013/job-fair.html

Thanks for the YAPC link. We're UK based though, so quite a way away from Texas. :(

We do online games (amongst other things). Though I don't want to go into too many details as I try to keep a degree of anonymity online.

Re: Security incident update

#279

Post by eridius »

stock_toaster wrote: post #276
eridius wrote: post #263

Earlier quoted context omitted.

I suppose they could also keep the decrypted key in memory using something like gpg-agent.

I read somewhere (can't remember where now) that the hacker supposedly dumped the key from memory. My guess is they type in the passphrase on boot or something, and it lives in an agent thereafter.

I haven't seen that anywhere, and I find it unlikely.

Re: Security incident update

#280

Post by scott_karana »

lcampbell wrote: post #252
scott_karana wrote: post #223

Earlier quoted context omitted.

Do you have a chatlog or citation? I'm not arguing, I'd love to read it! :)

Here's the writeup from seclists.org: http://seclists.org/nmap-dev/2013/q2/3 It looks like seclists.org and nmap.org are hosted on different IP addresses, but both route to linode, and both have the same WHOIS information, so I think it's safe to say the writeup applies to both.

Thanks!

By the sounds of their writeup, the attacks just visited the "highest profile" sites as per the Quora link to obfuscate their real attack.

Post reply on HN