Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Canon's cloud platform has lost users' files and can't restore them

digitalcameraworld.com

Post reply on HN
Print view View original

21–30 of 419 posts

Re: Canon's cloud platform has lost users' files and can't restore them

#21

Post by newyorker2 »

Dylan16807 wrote: post #15
SturgeonsLaw wrote: post #7

I take nightly images of my computer's primary drive and replicate them offsite, with alerting if it fails and automated scheduled backup restore testing. It blows my mind to think that some random schmuck like me has better backup procedures than these multibillion dollar global corporations. Companies that don't invest in their IT end up paying for it tenfold down the track.

But insufficient redundancy for that much data saves millions of dollars!

...which can instead be pocketed by higher ups in the company food chain :)

Re: Canon's cloud platform has lost users' files and can't restore them

#22

Post by syrgian »

londons_explore wrote: post #11
CalRobert wrote: post #5

Earlier quoted context omitted.

If memory serves, a class of Glacier storage (AWS) isn't practical to use in the EU because the inability to delete makes it impossible to serve "Right to Erasure" GDPR requests.

Store data and keys seperately. The set of keys for all your customers should be only a few megabytes, so is much much easier to back up more frequently and more copies of. It's also easier to have a "we do daily backups and delete any backups after 21 days" policy.

Wouldn't you need to edit every single backup where the key is stored in order to comply?

Re: Canon's cloud platform has lost users' files and can't restore them

#25

Post by Sharlin »

klodolph wrote: post #2

Lesson for those designing cold-storage solutions--design your system, as much as reasonable, to not support deletion operations. The reason why files get lost is usually due to software bugs, bad configurations, and operator errors. Design your system to protect against these things.

To be fair, image.canon is definitely not a cold-storage or backup solution, never mind primary storage. Its main use case is acting as a buffer for auto-uploading photos from camera and then later downloading them to a computer. As an afterthought it also offers a tiny long-term storage space, 10 GB per user, smaller than most memory cards available today.

Re: Canon's cloud platform has lost users' files and can't restore them

#26

Post by StreamBright »

radicaldreamer wrote: post #6

This is all because someone thought it would be “easy” to add a stream of recurring revenue for cloud photo storage but didn’t take the time to design the service from a technical perspective to be resilient enough to not crash and burn.

Or maybe an engineering mistake? I do not think you can claim anything without knowing the details. AWS lost customer data too, I would not try to assign blame without reading the post mortem.

Re: Canon's cloud platform has lost users' files and can't restore them

#27

Post by nikanj »

cbg0 wrote: post #17
radicaldreamer wrote: post #6

This is all because someone thought it would be “easy” to add a stream of recurring revenue for cloud photo storage but didn’t take the time to design the service from a technical perspective to be resilient enough to not crash and burn.

More likely than not some manager decided all those extra nines weren't really necessary.

The manager probably put five 9s onto the spec sheet, and provided absolutely no resources. You’ll never guess who’s going to get the heat for this, the manager or the poor person who didn’t follow the spec.

Re: Canon's cloud platform has lost users' files and can't restore them

#28

Post by Sharlin »

StreamBright wrote: post #26
radicaldreamer wrote: post #6

This is all because someone thought it would be “easy” to add a stream of recurring revenue for cloud photo storage but didn’t take the time to design the service from a technical perspective to be resilient enough to not crash and burn.

Or maybe an engineering mistake? I do not think you can claim anything without knowing the details. AWS lost customer data too, I would not try to assign blame without reading the post mortem.

According to TFA (which apparently nobody reads) it most certainly was an engineering mistake. Files in long-term storage were accidentally subject to the same code that deletes files in short-term (30 day) storage.

Re: Canon's cloud platform has lost users' files and can't restore them

#29

Post by MrGilbert »

dannyw wrote: post #20
SturgeonsLaw wrote: post #7

I take nightly images of my computer's primary drive and replicate them offsite, with alerting if it fails and automated scheduled backup restore testing. It blows my mind to think that some random schmuck like me has better backup procedures than these multibillion dollar global corporations. Companies that don't invest in their IT end up paying for it tenfold down the track.

On topic topic, can I get HN’s advice on my backup strategy? All important files on Synology Ds218+ NAS, which has 2x12TB helium HDDs in mirror. Daily HyperBackup to Google Drive. I test restoring files on a monthly basis. Email alerts on failure. Anything I can improve?

Looks like a sweet-spot to me.

Re: Canon's cloud platform has lost users' files and can't restore them

#30

Post by stingraycharles »

klodolph wrote: post #2

Lesson for those designing cold-storage solutions--design your system, as much as reasonable, to not support deletion operations. The reason why files get lost is usually due to software bugs, bad configurations, and operator errors. Design your system to protect against these things.

Yeah, it seems some of the other commenters miss that point: it’s not as if some disk crashed or something, it’s a bug in a software upgrade that deleted long term storage in addition to short term storage.

In my experience, the best way to mitigate these kind of things is to use PITR, which is always immutable. You make periodic snapshots, keep track of all changes since those snapshots. And then you use a policy to determine just how much data you want to keep here.

The problem is that not supporting delete operations is often very much impractical. Consider for example the recent heat Flickr (or was it Instagram?) has gotten for not physically deleting files but just marking them as deleted/invisible. You sometimes have really good reasons that you must support delete operations, so you must just make sure that there’s a reasonable delay before they are actually completely gone.

Post reply on HN