Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Total data loss after botched GitOps and failed backups

firefish.social

Post reply on HN
Print view View original

11–20 of 176 posts

Re: Total data loss after botched GitOps and failed backups

#11

Post by bg24 »

Very sad. But also a miss on admin's part, even if unfortunate. They did not realize during manual restore testing that volume was not being backed up.

"Yes and also apparently no. We use #Velero to capture backups of our cluster every 6 hours. From what I had seen our backups had been running successfully. I discovered once the incident started that backups had captured everything but the Persistent Volume Claim data. While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually. After digging into why our restores were not coming up with data, I found that our recurring backups were missing the flag to run volume backups with Restic which snapshots PVC block volume data."

Re: Total data loss after botched GitOps and failed backups

#13

Post by yowlingcat »

devmor wrote: post #9

Automation is wonderful for creating things, configuring things and moving things. Automation should never "clean up". Do your cleanup manually, or this is what you get.

Or said another way -- deletion is dangerous. I've observed that mixing the convenience of automation with the risk of hard deletion is fraught with peril. If you really need automation around deletion, it's best to set up roadblocks and use tombstoning approaches, where archive/backup data is sent to S3 Glacier or something of that ilk. Storage is cheap enough these days that there's no reason not to.

Re: Total data loss after botched GitOps and failed backups

#14

Post by scarface_74 »

The only data that I really care about are my photos and videos and they are backed up to four different providers.

But this also made me realize how upset I would be if my blog over at micro.blog was accidentally deleted. It’s more of a journal of my digital nomadding with my wife across the US than anything else.

I immediately went over and started a JSON export in bar (?) format.

Re: Total data loss after botched GitOps and failed backups

#15

Post by bagels »

stym06 wrote: post #12

That sucks! Now I'm thinking my github action that is doing a `git pull repo && cd repo/ && rm -rf dist/ && populate.sh && git push` in my bash script will totally go rogue one day and kill everything. Any idea, peeps?

Separate your code from your data. Be very careful with your data, and make sure you have backups of your data and that you test and validate your backup procedure. At the very least, verify the integrity of your data backups.

Re: Total data loss after botched GitOps and failed backups

#16

Post by ComputerGuru »

Thorrez wrote: post #10
readthenotes1 wrote: post #2

It's odd to me when people say backups are a feature. They are not. Restoration, however, is. And has to be tested...

It sounds like they did test restorations, but didn't correctly verify the results. It's not very clear though. >While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually.

They didn’t test the backups they were actually using, but a separate backup made completely separately to reassure themselves that .

Re: Total data loss after botched GitOps and failed backups

#17

Post by kuchenbecker »

pdntspa wrote: post #4

Wow... tooling that deletes a bunch of stuff because a manifest file is missing. Just... wow. I feel for the admin here. This seems equivalent to an accidental `sudo rm -rf /`.

Yeah, it's a hard lesson. We would always safe-expire or soft-delete to avoid this so each operation is undo-able for ~2 weeks.

Re: Total data loss after botched GitOps and failed backups

#18

Post by sp332 »

devmor wrote: post #9

Automation is wonderful for creating things, configuring things and moving things. Automation should never "clean up". Do your cleanup manually, or this is what you get.

The point of the tooling is that you describe what you want your deploy to look like, and it updates the deploy to match the description. If you delete something from the description and it stays running, that would be very confusing.

Re: Total data loss after botched GitOps and failed backups

#19

Post by markburns »

Thorrez wrote: post #10
readthenotes1 wrote: post #2

It's odd to me when people say backups are a feature. They are not. Restoration, however, is. And has to be tested...

It sounds like they did test restorations, but didn't correctly verify the results. It's not very clear though. >While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually.

I think they are saying they manually tested what they thought the automated process was doing, then found out their assumption about what it was doing was wrong. So the manual tests were moot.

Re: Total data loss after botched GitOps and failed backups

#20

Post by simula67 »

> While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually. After digging into why our restores were not coming up with data, I found that our recurring backups were missing the flag to run volume backups with Restic which snapshots PVC block volume data.

Can someone explain this? How did they test restores, if the actual restore failed to come up with data?

Post reply on HN