I'm a bit suprised that the affected machines were powered off instead of just disconnected. Would that not make an audit more complicated?
Security Incident on FreeBSD Infrastructure
11–20 of 37 posts
Re: Security Incident on FreeBSD Infrastructure
#12I'm a bit suprised that the affected machines were powered off instead of just disconnected. Would that not make an audit more complicated?
On the contrary -- you have to at least admit that if you disconnect a compromised machine, the attacker could have installed a script that detects that the machine is disconnected and erases evidence.
If you power off, you can always mount the hard drive read-only and do a forensic analysis.
Re: Security Incident on FreeBSD Infrastructure
#13I'm a bit suprised that the affected machines were powered off instead of just disconnected. Would that not make an audit more complicated?
Yes, it's more complicated than just SSH'ing into the server.
But on a compromised machine you can't trust anybody, not even the kernel. Assuming the worst, the attacker could have gained root privileges and modified the kernel or the base tools like ls and grep. You also can't trust the log files if they're not stored off-site. The modified kernel or ls could hide the attacker's traces from you.
Thus, the only possibility to really make sure nothing is hidden from you is to (power off the machine and) attach its hard disks to a trusted computer where they're mounted and investigated.
Re: Security Incident on FreeBSD Infrastructure
#14They use SVN still?
What's wrong with svn?
Re: Security Incident on FreeBSD Infrastructure
#15Interesting choice that some machines will not be reinstalled, only "thoroughly audited".
There are some systems (generally speaking, ones which were installed in the past few weeks) for which we know exactly what files should be installed and what their SHA256 hashes are. Thoroughly audited means "every single bit is correct".
Thanks for clarifying.
I assume if they had root though, they could theoretically install a rootkit in the MBR. Did you SHA256 verify the MBR too? :)
Re: Security Incident on FreeBSD Infrastructure
#16Earlier quoted context omitted.
There are some systems (generally speaking, ones which were installed in the past few weeks) for which we know exactly what files should be installed and what their SHA256 hashes are. Thoroughly audited means "every single bit is correct".
Thanks for clarifying. I assume if they had root though, they could theoretically install a rootkit in the MBR. Did you SHA256 verify the MBR too? :)
I didn't (I'm not involved in cluster management) but I suspect that it was done by someone.
Re: Security Incident on FreeBSD Infrastructure
#17They use SVN still?
What's wrong with svn?
Nothing, if you don't mind using a centralized development model.
The BSD's tend to be ultra-conservative in regard to version control systems - for example OpenBSD is still on CVS and developed their own version: http://www.opencvs.org
Re: Security Incident on FreeBSD Infrastructure
#18I'm a bit suprised that the affected machines were powered off instead of just disconnected. Would that not make an audit more complicated?
Yes, it's more complicated than just SSH'ing into the server. But on a compromised machine you can't trust anybody, not even the kernel. Assuming the worst, the attacker could have gained root privileges and modified the kernel or the base tools like ls and grep. You also can't trust the log files if they're not stored off-site. The modified kernel or ls could hide the attacker's traces from you. Thus, the only possi…
Live forensics are preferred to just reading disk; lots of problems with that and solutions to those problems on various types of machine.
Re: Security Incident on FreeBSD Infrastructure
#19I'm a bit suprised that the affected machines were powered off instead of just disconnected. Would that not make an audit more complicated?
On the contrary -- you have to at least admit that if you disconnect a compromised machine, the attacker could have installed a script that detects that the machine is disconnected and erases evidence. If you power off, you can always mount the hard drive read-only and do a forensic analysis.
On the contrary - you have to at least admit that if you power off a compromised machine, the attacker could have installed all his code in RAM only so that powering off erases evidence.
If you disconnect, you can still try to examine the current memory content and do a forensic analysis.
Of course what you really want is a memory dump via a trusted channel while the CPU is halted (hardware hypervisor or something like that) and then immediately power down. This is usually not supported on COTS hardware, so you have to choose the strategy that will erase the least evidence (power off, disconnect, suspend to disk, VM snapshot, whatever) depending on what you suspect the attack to be.
Re: Security Incident on FreeBSD Infrastructure
#20I'm a bit suprised that the affected machines were powered off instead of just disconnected. Would that not make an audit more complicated?
If you don't have the resources to take a snapshot of memory without disturbing the operation, then power off is the right thing to do.
For all you know, the trojan might uninstall itself or otherwise delete all evidence after a short while without network connectivity to a control center ("a dead man's switch").
Or, it patches the kernel so that it is "invisible" from the inside.
What you want to do is take a snapshot of the memory (from outside the kernel, if you can) and then power down.