41–50 of 419 posts
This is all because someone thought it would be “easy” to add a stream of recurring revenue for cloud photo storage but didn’t take the time to design the service from a technical perspective to be resilient enough to not crash and burn.
Or maybe an engineering mistake? I do not think you can claim anything without knowing the details. AWS lost customer data too, I would not try to assign blame without reading the post mortem.
Lesson for those designing cold-storage solutions--design your system, as much as reasonable, to not support deletion operations. The reason why files get lost is usually due to software bugs, bad configurations, and operator errors. Design your system to protect against these things.
Yeah, it seems some of the other commenters miss that point: it’s not as if some disk crashed or something, it’s a bug in a software upgrade that deleted long term storage in addition to short term storage. In my experience, the best way to mitigate these kind of things is to use PITR, which is always immutable. You make periodic snapshots, keep track of all changes since those snapshots. And then you use a policy to…
Earlier quoted context omitted.
No. Losing data is just as bad from a GDPR perspective.
Oh, really? Can you actually be fined for destroying disks containing customer information?
This is all because someone thought it would be “easy” to add a stream of recurring revenue for cloud photo storage but didn’t take the time to design the service from a technical perspective to be resilient enough to not crash and burn.
My two cents is going to be that the ad tech MacBook deleted the old S3 bucket for mockup but it wasn’t a mockup at that point
Lesson for those designing cold-storage solutions--design your system, as much as reasonable, to not support deletion operations. The reason why files get lost is usually due to software bugs, bad configurations, and operator errors. Design your system to protect against these things.
I can't access the article, but there was a recent ransomware attack on Canon [1]. I wonder if this could be related to it. [1] https://news.ycombinator.com/item?id=24185734
The timing is convenient though.
This is all because someone thought it would be “easy” to add a stream of recurring revenue for cloud photo storage but didn’t take the time to design the service from a technical perspective to be resilient enough to not crash and burn.
Incredible damage for some users can be negligible damage to the company. In this case just 10TB of data was lost, so maybe thousands of users for the long term storage option.
Losing customer data has potential for big reputation or brand damage, but surprisingly often the damage is relatively small. Putting in too much effort when potential damage to business is minimal may not be worth of the cost.
I take nightly images of my computer's primary drive and replicate them offsite, with alerting if it fails and automated scheduled backup restore testing. It blows my mind to think that some random schmuck like me has better backup procedures than these multibillion dollar global corporations. Companies that don't invest in their IT end up paying for it tenfold down the track.
On topic topic, can I get HN’s advice on my backup strategy? All important files on Synology Ds218+ NAS, which has 2x12TB helium HDDs in mirror. Daily HyperBackup to Google Drive. I test restoring files on a monthly basis. Email alerts on failure. Anything I can improve?