What's really important is that you never should keep a compromised system like this running, even if you think you found all modifications the attacker did. You probably didn't. So save your configs and set this machine up from scratch.
[deleted]
Hacked. A Short Story
31–40 of 50 posts
Re: Hacked. A Short Story
#32What's really important is that you never should keep a compromised system like this running, even if you think you found all modifications the attacker did. You probably didn't. So save your configs and set this machine up from scratch.
Save any data you need and consider the machine totally compromised.
By that, I mean you need to consider any IPMI processor compromised as well, since these can be attacked (or their secrets stolen). Basically rebuild the machine from as low a level as you can. Depending on the brand of server hardware, you may need physical access.
Re: Hacked. A Short Story
#33Stories like this is what makes me believe immutable infrastructure is the future.
curious to know how you can make a linux distro immutable from any modification.
Build a custom LiveISO.
There are plenty of tutorials, but it is boring to follow them manually, especially if you need to make a couple of tweaks to ISO or update it.
So I put together a couple of scripts to automate the process:
Re: Hacked. A Short Story
#34What's really important is that you never should keep a compromised system like this running, even if you think you found all modifications the attacker did. You probably didn't. So save your configs and set this machine up from scratch.
At first I felt bad that I didn't know all the tricks the author knew, but then I realized I would never, ever try to clean a machine. I would copy (and verify) and config files I needed and just spin up a new vm. Its not worth the time and there's a slim chance you'll actually catch everything. Funny how he didn't just run rootkit hunter. I wonder if it would have detected this hack.
I'm also starting to think that the move from Linux being a sysadmin-only OS to being something that can casually be spun up by even the most jr of developers, means we're having a Microsoft circa 1997 moment where security needs to be commoditized. I wonder if any of the few commercial AV's for Linux are updated frequently enough to catch hacks like these.
Re: Hacked. A Short Story
#35What's really important is that you never should keep a compromised system like this running, even if you think you found all modifications the attacker did. You probably didn't. So save your configs and set this machine up from scratch.
At first I felt bad that I didn't know all the tricks the author knew, but then I realized I would never, ever try to clean a machine. I would copy (and verify) and config files I needed and just spin up a new vm. Its not worth the time and there's a slim chance you'll actually catch everything. Funny how he didn't just run rootkit hunter. I wonder if it would have detected this hack. I'm also starting to think that…
I think he's too clever for his own good. This machine should have been considered beyond salvage, remove any precious data (you did have a backup, didn't you?) and re-image. I've had to recover data from hacked boxes a few times for 'brand new customers' and the first thing I do with a system like that is to make sure I get a console wired up and the uplink disconnected. No point in taking chances.
Re: Hacked. A Short Story
#36What's really important is that you never should keep a compromised system like this running, even if you think you found all modifications the attacker did. You probably didn't. So save your configs and set this machine up from scratch.
This is correct, of course, in principle, but in the real world its often not feasible. As a freelancer, the author likely spends much of his time working on small business blogs and brochure sites.
These are frequently set up on a single server, containing code, database, and any file storage necessary. They're frequently written by cloning and modifying a framework, making reinstalling from a known good copy difficult to impossible. They often have no version control and are frequently configured to allow the webserver to modify code, so there is rarely a known good copy of the site code. In short, what's running on the compromised server is often the only usable iteration of the site's code available.
I hope nobody on HN would willingly set up a site like this, but the fact remains that they exist and need to be maintained. Attempting to clean a system in place is nowhere near 100% effective, but it's much more effective than doing nothing, which is exactly what will happen if you try to tell a client that they have to be offline for days, lose several months of data, and pay someone to rewrite half their site from scratch because you want to format their server and attempt to rebuild it.
Re: Hacked. A Short Story
#37Offtopic: blog headers are becoming larger by the year it seems. Have there been any studies to see what such large headers do to readership?
Splash pages are back [1]
[1] http://www.cosmonautdreams.com/html/blog/websites/readabilit...
Re: Hacked. A Short Story
#38Earlier quoted context omitted.
Still, he provided a service that was important for the customer. And, the customer wanted him to configure exim4 because a lot of the emails sent were marked as spam. The compromise was most likely the biggest reason for that.
Sometimes it's like that. He could try and get the customer to pay him for this but he might lose on the long term if they've been a recurring customer for a long time.
Let's be honest.
The moment he discovered an attacker he should have been on the phone with the client explaining the situation and asking how to proceed. If the client wanted him to shut down and clean up the hack, he should be charging for that. If not, he should have done nothing.
Transparency resolves most problems.
Re: Hacked. A Short Story
#39What's really important is that you never should keep a compromised system like this running, even if you think you found all modifications the attacker did. You probably didn't. So save your configs and set this machine up from scratch.
I was gonna say much the same thing. Once a machine has been compromised, you've got to format and re-image it. The machine in the article could well still have a rootkit installed. Not to mention he doesn't seem to know the initial attack vector, so any security updates are meaningless if someones credentials have been stolen. Finally, if you're really paranoid, you also have to flash the bios in the machine too as…
This seems like a great reason to virtualize everything possible.
Re: Hacked. A Short Story
#40What's really important is that you never should keep a compromised system like this running, even if you think you found all modifications the attacker did. You probably didn't. So save your configs and set this machine up from scratch.
This is correct, of course, in principle, but in the real world its often not feasible. As a freelancer, the author likely spends much of his time working on small business blogs and brochure sites. These are frequently set up on a single server, containing code, database, and any file storage necessary. They're frequently written by cloning and modifying a framework, making reinstalling from a known good copy diffic…
If moving the project to a new server is not feasible, then how does the client backup their data?
I refuse to work on projects that cannot be installed from scratch in a local VM.