Would be way cooler if he rooted 2600 servers ;)
Former Hostgator employee arrested, charged with rooting 2,700 servers
41–50 of 61 posts
Re: Former Hostgator employee arrested, charged with rooting 2,700 servers
#42Wow, he actually patched 'netstat' and 'ps'. It must have been to hide certain processes and port numbers from showing up. I wonder how may one go about understanding if they are using a 'hacked' version of something as non-trivial and comprehensive as 'ps' and 'netstat'. I mean, if I'm not suspicious, I wouldn't give it a second though and trust the output of these commands.
Ideally, you should be running some kind of tripwire scan. If the hash for common utilities changes and you didn't update those binaries, then something bad has happened. Hacked versions of common utilities is a common payload for rootkits.
If you lose superuser, you've almost certainly lost your kernel, at which point your only hope for Tripwire-type scans actually working is an attacker that doesn't know where to download a good rootkit from.
You should still run the scans, just be aware of the limitations.
Re: Former Hostgator employee arrested, charged with rooting 2,700 servers
#43Did these 2700 servers play a role in any DDoS attacks as well? It would be quite a lucrative stance for the employee to sell access to these servers to one or more groups who could potentially make more use of them.
2700 servers all on the same network makes for far less of a DDoS attack then 2700 similar servers on different networks - and it's far easier to detect and block too.
They would be more valuable for bitcoin mining most likely.
Re: Former Hostgator employee arrested, charged with rooting 2,700 servers
#44Earlier quoted context omitted.
It's fairly expensive (processing and storage), but it's well worth it for secure environments. I've worked for companies who have this set up on their Windows Server environment (since they were administered through the remote GUI) and SSH logging for the Unix/Linux servers when running as root.
Expensive? 1 image a minute for a 40 hour work week is 2,400 images or 120k a year. We're talking screenshots so they aren't large files (you don't need a high bit rate), probably in the range of 500KB which would mean a whole year is less than 60GB. 3TB hard drives cost ~$130 these days and have room to store 50 employee years of screenshots. If money is that tight you can always compress the images and get double o…
"3TB hard drives cost ~$130 these days and have room to store … "
No trying to call out jinknee specifically, but I often see this argument:
"But storage only costs ~$50/TB!",
and I read:
"here's someone who's got no idea - who's never seen what 'enterprise' pays for proper secured/redundant/backed-up/auditable/managed storage."
Does anyone _really_ think sending the IT department junior round to BestBuy to grab ~$400 worth of external USB drives would then let you say "Right, that's then next 5 years worth of storage and archive of high-security and potentially-lawsuit-relevant employee data sorted!"?
Re: Former Hostgator employee arrested, charged with rooting 2,700 servers
#45Earlier quoted context omitted.
It's fairly expensive (processing and storage), but it's well worth it for secure environments. I've worked for companies who have this set up on their Windows Server environment (since they were administered through the remote GUI) and SSH logging for the Unix/Linux servers when running as root.
Expensive? 1 image a minute for a 40 hour work week is 2,400 images or 120k a year. We're talking screenshots so they aren't large files (you don't need a high bit rate), probably in the range of 500KB which would mean a whole year is less than 60GB. 3TB hard drives cost ~$130 these days and have room to store 50 employee years of screenshots. If money is that tight you can always compress the images and get double o…
[deleted]
Re: Former Hostgator employee arrested, charged with rooting 2,700 servers
#46Wow, he actually patched 'netstat' and 'ps'. It must have been to hide certain processes and port numbers from showing up. I wonder how may one go about understanding if they are using a 'hacked' version of something as non-trivial and comprehensive as 'ps' and 'netstat'. I mean, if I'm not suspicious, I wouldn't give it a second though and trust the output of these commands.
It's been some time now, but I've been there...
Once you've lost root to a sufficiently competent attacker you can't trust _anything_ on that box any more.
One thing that'll help against (some) script kiddies with rootkits is to have available statically linked copies of every tool you might want to use to see what's happening on your box - for a long while, every colo-ed box I managed had a cdrom drive with read-only versions of /bin, /sbin, and useful bits of /usr - all with the binaries statically linked. They can in handy a few times (mostly to confirm that "yep, we're screwed. Get this box off the network and powered down immediately and implement the bring-up-a-new-server-from-scratch plan right now".
At some stage though, you can't trust the kernel or the hardware - an attacker who's got into your booted kernel or your bios or your network card firmware, if they're good enough, they probably can't be detected by examining anything you could see logged into the box itself. The only way to identify that level of attack is by monitoring the traffic from the box from some trusted piece of network gear upstream of your rooted server (and against a sufficiently talented attacker, even identifying unexpected outbound traffic might be impossible. If your list of likely attackers includes three letter agencies or nation states, I hope your getting your secutiry advice from somethere other than HN comments…)
Re: Former Hostgator employee arrested, charged with rooting 2,700 servers
#47I'm not surprised with that company actually, hostgator is a joke and the employees in Austin, Texas are too. I think the one thing that bothers me how did he get the SSH key? The fact that he had it tells me that someone higher up dropped the ball somewhere.
Re: Former Hostgator employee arrested, charged with rooting 2,700 servers
#48Earlier quoted context omitted.
Anything that lives outside of userspace. If one has root and patience - flash BIOS and wait for a coldboot. One can even get an IP stack to pull down new firmware between boots. The user sees a normal post screen and your hypervisor sees normal hardware adapters. See Jonathan Brossard's 'prior work' slide from his Defcon talk on his work [1] for more details on the state of X86 backdooring. 'Trusted computing' and a…
I know how they work, I'm asking GP to name one that he could have used. There isn't actually much "out there" that is undetectable. You'd likely have to write your own, which is highly non-trivial.
Obviously undetectable isn't true, but i'd guess that kbeast is likely to go undetected in all but the most prepared operations. Forensics shouldn't count since you already know you're fucked by then.
Re: Former Hostgator employee arrested, charged with rooting 2,700 servers
#49"Gisse didn't return a voicemail and e-mail seeking comment for this report. A Court docket shows he is scheduled to be arraigned next month and gives no indication he has entered a plea in the case. He's being held at the Harris County Jail on $20,000 bond, a spokeswoman at the district attorney's office said."
Wait, wait, just because the guy's in jail is no reason not to return voicemail and emails!
Re: Former Hostgator employee arrested, charged with rooting 2,700 servers
#50Wow, he actually patched 'netstat' and 'ps'. It must have been to hide certain processes and port numbers from showing up. I wonder how may one go about understanding if they are using a 'hacked' version of something as non-trivial and comprehensive as 'ps' and 'netstat'. I mean, if I'm not suspicious, I wouldn't give it a second though and trust the output of these commands.
It's been some time now, but I've been there... Once you've lost root to a sufficiently competent attacker you can't trust _anything_ on that box any more. One thing that'll help against (some) script kiddies with rootkits is to have available statically linked copies of every tool you might want to use to see what's happening on your box - for a long while, every colo-ed box I managed had a cdrom drive with read-onl…
Read only media is good advice and low effort compared to most viable defense. Certainly valuable at times worth doing. Starting from the assumption you'll get beaten typically pays off over perfecting the hack-proof nirvana.
Nobody should lose any sleep about BIOS embedding and similar - that level of attacker and sponsor imply a level of threat that no typical organization has a chance against.
In my opinion, after years of pondering dozens of intrusions with many types of ways in and regular failure of all kinds of defenses I don't think there is much advice to give aside from the flaw is in your custom software, stupid.
I have become a really big advocate of CM and push button provisioning with identical replacement hosts that build from scratch and commonly get refreshed - relying on code and configurations managed centrally.
The best way to remove an attacker is a complete rebuild. If you're already using Chef etc why not just dump them proactively? Some roles don't lend themselves to this, but I assure you it is like a massive weight has been lifted.