41–50 of 81 posts
Earlier quoted context omitted.
the 1 in the 3-2-1 should be somewhere on premise or at least not directly reachable from the internet. Think: ssh cron job that copies backups from cloud to cold storage
If the data you’re reading is encrypted, you’re still screwed.
(I mean, on large deltas anywhere)
Earlier quoted context omitted.
Backups need an air gap.
If they are not incremental but append only, an air gap is not strictly needed and can be used as an additional safeguard performed less frequently because of manual overhead. The crux of the matter is to assume the main system has been compromised and preventing overwriting existing data.
To utterly destroy an organisation you don't erase or encrypt their data. You change it. Slowly. A little by a little. A birthday here, a name there, a number ... Using the normal ways to change this data. In this way you can go undiscovered for years, employees get blamed for making stupid errors for a LONG time and there is absolutely no way to fix things, no matter what the backup strategy is.
Earlier quoted context omitted.
> It just means they have inept IT teams. More often than not in my experience is that the IT team wants proper backups but management baulk at the price and never authorize it. Until something bad happens of course.
And when the bad thing happens it's of course the IT team painted as "those guys who kept on bitching also earlier about their jobs"
What happened?
It is our best estimate that when servers had to be moved from one data center to another and despite the fact that the machines being moved were protected by both firewall and antivirus, some of the machines were infected before the move, with an infection that had not been actively used in the previous data center, and we had no knowledge that there was an infection.
During the work of moving servers from one data center to the other, servers that were previously on separate networks were unfortunately wired to access our internal network that is used to manage all of our servers.
Via the internal network, the attackers gained access to central administration systems and the backup systems.Shoddy journalism strikes again. How do we know that all data was lost and the notice on the homepage was not uploaded by hackers? "CloudNordic could not be reached for comment." It's a journalist's job to reach either the company or affected customers to verify the facts.
I don't think we need fearmongering about "shoddy journalism" for something so easy to check.
Edit: in Danish not Norwegian, my apologies
Unfortunate turn of events. I find it really hard to have empathy for serious businesses who don’t have backups and are dependent on a single cloud. Like for example if you are all in on AWS and do all your backups of your AWS systems to AWS then lose your account. Meh… your fault. If you run a business then you have an absolute obligation to be able to instantly bring your business back up outside your primary hosti…
At this standard, many small online businesses would not be possible. I think that would be a loss.
The solution to ransomware? Backups. It's not more complicated than that. It's honestly puzzling that ransomware is the issue it is, crippling entire organizations. It just means they have inept IT teams. Sucks this Danish cloud host provider didn't back stuff up properly.
maybe they were backing up their stuff properly, but backups were wiped as well. even if you have some fancy append-only storage someone still has access to it and that access can be misued.
Earlier quoted context omitted.
And what if the backup you're copying to cold storage is also encrypted? How did the saying go? You don't have backups until you've successfully restored from them or something like that. =) Basically any 3-2-1 system is Schrödinger's backup until you've actually used it.
So you only have 1 backup that you daily overwrite?
The parent comment is intending to remind people that many things can happen to a backup after it's done. Backups cannot be "set and forget", as just making the backup isn't enough since so many things can happen after you've taken that backup.
- Bitrot/bitflips silently corrupt your backups and your filesystem doesn't catch it
- The storage your backups are on goes bad suddenly before you can recover
- Your storage provider closes up shop suddenly or the services go down completely, etc
- malicious actors intentionally infiltrate and now your data is held hostage
- Some sysadmin accidentally nukes the storage device holding the backups or some other mistake (to summon the classic, I'm betting there are a few persons who have stories where an admin trying to clean up some leftover .temp files accidentally hit SHIFT while typing
```rm -rf /somedirectory/.temp```
and instead writes:
```rm -rf /somedirectory/>temp```
- (for image level backups) The OS was actually in a bad state/was infected, so even if you do restore the machine, the machine is in an unusable state
- A fault in the backup system results in garbage data being written to the backup "successfully" (If you're a VMware administrator and you got hit by a CBT corruption bug, you know what I'm talking about. If you aren't look just search VMware CBT and imagine that this system screws up and starts returning garbage data instead of the correct and actual changed blocks that the backup application was expecting)
Basically, unless you're regularly testing your backups, there isn't really any assurance that the data that was successfully written at the time of backup is still the same. Most modern backup programs have in-flight CRC checks to ensure that at the time of the backup, the data read from source is the same going into the backup, but this only confirms that the data integrity is stable at the time of the backup.
Many backup suites have "backup health checks" which can ensure the backup file integrity, but again, a successful test only means "at the time you ran the test, it was "okay". Such tests _still_ don't tell you whether or not the data in the backup file is actually usable/not compromised, it only tells you that the backup application confirms the data in the backup right now is the same as when the backup was first created.
So the parent post is correct; until you have tested your backups properly, you can't really be sure if your backups are worth anything.
Combine this with the fact that many companies handle backups very badly (no redundant copies, storing the backups directly with production data, relying only on snapshots, etc), and you end up with situations like in the article where a single ransomware attack takes down entire businesses.
Shoddy journalism strikes again. How do we know that all data was lost and the notice on the homepage was not uploaded by hackers? "CloudNordic could not be reached for comment." It's a journalist's job to reach either the company or affected customers to verify the facts.
Because CloudNordic says so on their temporary webpage. https://www.cloudnordic.com/ (in Danish, I used Google Translate to check and the result seems fine) I don't think we need fearmongering about "shoddy journalism" for something so easy to check. Edit: in Danish not Norwegian, my apologies
There is no excuse for this.
Our industry is mostly run by clowns and unserious people.