Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Former Hostgator employee arrested, charged with rooting 2,700 servers

arstechnica.com

Post reply on HN
Print view View original

21–30 of 61 posts

Re: Former Hostgator employee arrested, charged with rooting 2,700 servers

#21

Post by jonknee »

freehunter wrote: post #3
ghshephard wrote: post #2

This all seemed like a pretty run of the mill story about an insider violating company trust, and then getting caught - until the final sentence: "Among other things, a desktop monitoring system that took screenshots of employee workstations in one-minute increments helped Hostgator officials quickly zero in on Gisse." Not something I'd want on my personal system, but it's exactly the sort of thing that I think every…

It's fairly expensive (processing and storage), but it's well worth it for secure environments. I've worked for companies who have this set up on their Windows Server environment (since they were administered through the remote GUI) and SSH logging for the Unix/Linux servers when running as root.

Expensive? 1 image a minute for a 40 hour work week is 2,400 images or 120k a year. We're talking screenshots so they aren't large files (you don't need a high bit rate), probably in the range of 500KB which would mean a whole year is less than 60GB. 3TB hard drives cost ~$130 these days and have room to store 50 employee years of screenshots. If money is that tight you can always compress the images and get double or more for your cash.

As for processing, you need to find a new computer if you can notice a screenshot being taken.

Re: Former Hostgator employee arrested, charged with rooting 2,700 servers

#22

Post by homosaur »

ghshephard wrote: post #2

This all seemed like a pretty run of the mill story about an insider violating company trust, and then getting caught - until the final sentence: "Among other things, a desktop monitoring system that took screenshots of employee workstations in one-minute increments helped Hostgator officials quickly zero in on Gisse." Not something I'd want on my personal system, but it's exactly the sort of thing that I think every…

I don't think this is appropriate for general office use simply because you might leach private data from employees (bank accounts, retirement accounts, medical records, whatever), but in secure, restricted environments I think it's totally appropriate and probably needed.

Re: Former Hostgator employee arrested, charged with rooting 2,700 servers

#23

Post by homosaur »

tracker1 wrote: post #7
jamescun wrote: post #4

> While his root access gave Gisse access to private data stored on a large number of customer websites, there's no evidence he used it, the Hostgator executive said. I think the article is quick to jump to the conclusion that he was attempting to be malicious with his actions however this could be a case of Hanlon's razor. His actions could easily be attributed to a less-than-aware sysadmin developing his own soluti…

If he hasn't accessed any of these systems since he was terminated, he could state that it was for "emergency" access to remote systems upon other compromises. Since most of these systems are likely headless, then remote access is the only way to get in. A lot of remote exploits will nuke SSH, and other access tools, so having a "backdoor" is often a good idea. That said, it's still likely that this guy is just a dou…

I don't think this would work. Installing your own solution on customer facing hardware? Might get you out of prosecution, but I think you'd have a nearly impossible time explaining that.

Re: Former Hostgator employee arrested, charged with rooting 2,700 servers

#24

Post by __david__ »

ultimoo wrote: post #12

Wow, he actually patched 'netstat' and 'ps'. It must have been to hide certain processes and port numbers from showing up. I wonder how may one go about understanding if they are using a 'hacked' version of something as non-trivial and comprehensive as 'ps' and 'netstat'. I mean, if I'm not suspicious, I wouldn't give it a second though and trust the output of these commands.

> I wonder how may one go about understanding if they are using a 'hacked' version of something as non-trivial and comprehensive as 'ps' and 'netstat'.

So this happened to me once, but it wasn't ps or netstat but sshd that had been replaced. We only noticed it because the replacement sshd was bad and didn't set up the PATH correctly so "darcs push" stopped working (it was presumably set up for a different linux distro than we were running).

Re: Former Hostgator employee arrested, charged with rooting 2,700 servers

#25

Post by drzaiusapelord »

ultimoo wrote: post #12

Wow, he actually patched 'netstat' and 'ps'. It must have been to hide certain processes and port numbers from showing up. I wonder how may one go about understanding if they are using a 'hacked' version of something as non-trivial and comprehensive as 'ps' and 'netstat'. I mean, if I'm not suspicious, I wouldn't give it a second though and trust the output of these commands.

Ideally, you should be running some kind of tripwire scan. If the hash for common utilities changes and you didn't update those binaries, then something bad has happened.

Hacked versions of common utilities is a common payload for rootkits.

Re: Former Hostgator employee arrested, charged with rooting 2,700 servers

#26

Post by ttrreeww »

ultimoo wrote: post #12

Wow, he actually patched 'netstat' and 'ps'. It must have been to hide certain processes and port numbers from showing up. I wonder how may one go about understanding if they are using a 'hacked' version of something as non-trivial and comprehensive as 'ps' and 'netstat'. I mean, if I'm not suspicious, I wouldn't give it a second though and trust the output of these commands.

Amateur hour, quite a few rootkits out there are almost completely undetectable...

Re: Former Hostgator employee arrested, charged with rooting 2,700 servers

#27

Post by saurik »

ultimoo wrote: post #17
saurik wrote: post #15

Earlier quoted context omitted.

While this isn't guaranteed (all tools, including the compiler, may be patched), you can use checks and balances: verify /proc doesn't contain phantom processes, compile your own copy of ps, try more-obscure tools like top. If by "understand" you just mean "notice"... well, you don't, until one day you accidentally stumble across one of the above and start digging. (Maybe, for example, you install some kind of server…

Nice. So both times that you noticed this, was it malice on someone's part due to which the tools were patched? I have just never heard or encountered such a situation and am frankly paranoid about something like this happening to one of the tools I use.

I've been pwned with 0-days in various email servers: sendmail over a decade ago, and exim4 more recently (still many years ago, though). The patched copy of ssh on one of my boxes was then distributing passwords to someone, and which then was used to gain access to another machine.

What I'm always paranoid about is that I work in a community of security researchers that sit on and occasionally drop 0-days: I have very little trust that much software is actually remotely "secure". Meanwhile, the only reason I had noticed those other attacks is just how sloppy they were... a more targeted-to-me run by a more careful attacker would have maybe never been noticed.

It has drastically changed the way I think about security, FWIW; as one example: I don't every store logs on a box being logged anymore. Instead, logs are immediately transported to another machine whose only purpose is to accept and store logs (and so is listening for incoming log packets, OpenSSH, and nothing else. The first thing anyone does is attempt to patch themselves out of logs (one attack I noticed because wtmp was mysteriously damaged).

Re: Former Hostgator employee arrested, charged with rooting 2,700 servers

#28

Post by freehunter »

jonknee wrote: post #21
freehunter wrote: post #3

Earlier quoted context omitted.

It's fairly expensive (processing and storage), but it's well worth it for secure environments. I've worked for companies who have this set up on their Windows Server environment (since they were administered through the remote GUI) and SSH logging for the Unix/Linux servers when running as root.

Expensive? 1 image a minute for a 40 hour work week is 2,400 images or 120k a year. We're talking screenshots so they aren't large files (you don't need a high bit rate), probably in the range of 500KB which would mean a whole year is less than 60GB. 3TB hard drives cost ~$130 these days and have room to store 50 employee years of screenshots. If money is that tight you can always compress the images and get double o…

If you're talking bare minimum, yeah. The software solution this company had wasn't quite that barebones. It was a robust suite, and actually did take some planning on the desktop and server side to set aside resources for it.

The software was part of a package that managed software installs as well.

Re: Former Hostgator employee arrested, charged with rooting 2,700 servers

#29

Post by lawnchair_larry »

ghshephard wrote: post #2

This all seemed like a pretty run of the mill story about an insider violating company trust, and then getting caught - until the final sentence: "Among other things, a desktop monitoring system that took screenshots of employee workstations in one-minute increments helped Hostgator officials quickly zero in on Gisse." Not something I'd want on my personal system, but it's exactly the sort of thing that I think every…

I'll never work for any employer who does that, so I'd have trouble calling it a reasonable thing to do.

Re: Former Hostgator employee arrested, charged with rooting 2,700 servers

#30

Post by davorak »

lawnchair_larry wrote: post #29
ghshephard wrote: post #2

This all seemed like a pretty run of the mill story about an insider violating company trust, and then getting caught - until the final sentence: "Among other things, a desktop monitoring system that took screenshots of employee workstations in one-minute increments helped Hostgator officials quickly zero in on Gisse." Not something I'd want on my personal system, but it's exactly the sort of thing that I think every…

I'll never work for any employer who does that, so I'd have trouble calling it a reasonable thing to do.

What alternative do you think is reasonable to help track down problems like this after the fact?
Post reply on HN