Canon's cloud platform has lost users' files and can't restore them • Hacker News Board

Re: Canon's cloud platform has lost users' files and can't restore them

#61

Post by ChuckNorris89 » Tue, Aug 18, 2020, 9:05 AM UTC

nabla9 wrote: post #49

radicaldreamer wrote: post #6

This is all because someone thought it would be “easy” to add a stream of recurring revenue for cloud photo storage but didn’t take the time to design the service from a technical perspective to be resilient enough to not crash and burn.

This can be just calculated risk. Incredible damage for some users can be negligible damage to the company. In this case just 10TB of data was lost, so maybe thousands of users for the long term storage option. Losing customer data has potential for big reputation or brand damage, but surprisingly often the damage is relatively small. Putting in too much effort when potential damage to business is minimal may not be…

This. Outside of FAANG, Gov and finance, very few consumer facing companies have the budgets to engineer redundancies.

Re: Canon's cloud platform has lost users' files and can't restore them

#62

Post by kohtatsu » Tue, Aug 18, 2020, 9:07 AM UTC

nabla9 wrote: post #49

radicaldreamer wrote: post #6

This is all because someone thought it would be “easy” to add a stream of recurring revenue for cloud photo storage but didn’t take the time to design the service from a technical perspective to be resilient enough to not crash and burn.

This can be just calculated risk. Incredible damage for some users can be negligible damage to the company. In this case just 10TB of data was lost, so maybe thousands of users for the long term storage option. Losing customer data has potential for big reputation or brand damage, but surprisingly often the damage is relatively small. Putting in too much effort when potential damage to business is minimal may not be…

This comment reads as "This is normal; nothing to see here move along".

Which is more of an indictment on the state of things than on the comment itself.

If you can't protect people's shit, don't offer to hold onto it for money!

Re: Canon's cloud platform has lost users' files and can't restore them

#63

Post by visarga » Tue, Aug 18, 2020, 9:07 AM UTC

chrismatheson wrote: post #59

mixedbit wrote: post #46

Earlier quoted context omitted.

Both AWS S3 and Google cloud storage buckets have an option that makes it impossible to delete stored objects for some period of time. The option was added for some legal compliance reasons, but I find it useful as an extra safeguard that important service data is not accidentally or maliciously deleted.

I suppose that feature would be incompatible with user data in a GDPR world ??

Git is also incompatible with GDPR, you can't simply delete a file from all history.

Re: Canon's cloud platform has lost users' files and can't restore them

#64

Post by dodobirdlord » Tue, Aug 18, 2020, 9:08 AM UTC

syrgian wrote: post #22

londons_explore wrote: post #11

Earlier quoted context omitted.

Store data and keys seperately. The set of keys for all your customers should be only a few megabytes, so is much much easier to back up more frequently and more copies of. It's also easier to have a "we do daily backups and delete any backups after 21 days" policy.

Wouldn't you need to edit every single backup where the key is stored in order to comply?

I assume they mean the standard practice of encrypting your backups with per-customer encryption keys. If you ever want to destroy all of the data for a particular customer you don't have to do any editing of backups, you just have to nuke every copy of the encryption key you use for that customer.

Re: Canon's cloud platform has lost users' files and can't restore them

#65

Post by tyfon » Tue, Aug 18, 2020, 9:08 AM UTC

chrismatheson wrote: post #59

mixedbit wrote: post #46

Earlier quoted context omitted.

Both AWS S3 and Google cloud storage buckets have an option that makes it impossible to delete stored objects for some period of time. The option was added for some legal compliance reasons, but I find it useful as an extra safeguard that important service data is not accidentally or maliciously deleted.

I suppose that feature would be incompatible with user data in a GDPR world ??

Depends on how long it is stored, you have some time (one month) to clear your systems when you get a request for deletion.

Re: Canon's cloud platform has lost users' files and can't restore them

#66

Post by hellofunk » Tue, Aug 18, 2020, 9:08 AM UTC

There is one somewhat reassuring lesson here -- No matter the company, no matter the brand, no matter their worth: software development is wrought with challenges, and major mistakes for seemingly obvious things are made by everyone, all the time, at every level.

Re: Canon's cloud platform has lost users' files and can't restore them

#67

Post by perlgeek » Tue, Aug 18, 2020, 9:08 AM UTC

nabla9 wrote: post #49

radicaldreamer wrote: post #6

This is all because someone thought it would be “easy” to add a stream of recurring revenue for cloud photo storage but didn’t take the time to design the service from a technical perspective to be resilient enough to not crash and burn.

This can be just calculated risk. Incredible damage for some users can be negligible damage to the company. In this case just 10TB of data was lost, so maybe thousands of users for the long term storage option. Losing customer data has potential for big reputation or brand damage, but surprisingly often the damage is relatively small. Putting in too much effort when potential damage to business is minimal may not be…

It's easy for this calculated risk to misfire, if the platform gets a reputation for losing data early in its life.

Re: Canon's cloud platform has lost users' files and can't restore them

#68

Post by hendry » Tue, Aug 18, 2020, 9:11 AM UTC

I was wondering what those weird messages from https://image.canon/ were about.

IIUC the idea of their new platform was to be ephemeral in any case. Just to give a pipeline to stream in to other accounts.

I had no idea it was offering long term storage. Better off with S3 Glacier for that!

Re: Canon's cloud platform has lost users' files and can't restore them

#69

Post by nix23 » Tue, Aug 18, 2020, 9:11 AM UTC

ChuckNorris89 wrote: post #61

nabla9 wrote: post #49

Earlier quoted context omitted.

This can be just calculated risk. Incredible damage for some users can be negligible damage to the company. In this case just 10TB of data was lost, so maybe thousands of users for the long term storage option. Losing customer data has potential for big reputation or brand damage, but surprisingly often the damage is relatively small. Putting in too much effort when potential damage to business is minimal may not be…

This. Outside of FAANG, Gov and finance, very few consumer facing companies have the budgets to engineer redundancies.

>budgets to engineer redundancies

You dont need that, but a backup ;)

Re: Canon's cloud platform has lost users' files and can't restore them

#70

Post by mixedbit » Tue, Aug 18, 2020, 9:11 AM UTC

chrismatheson wrote: post #59

mixedbit wrote: post #46

Earlier quoted context omitted.

Both AWS S3 and Google cloud storage buckets have an option that makes it impossible to delete stored objects for some period of time. The option was added for some legal compliance reasons, but I find it useful as an extra safeguard that important service data is not accidentally or maliciously deleted.

I suppose that feature would be incompatible with user data in a GDPR world ??

I'm not a lawyer, but my understanding is that GDPR requires companies to remove user data upon request in reasonable time-frame. If you keep not deletable backups for one month in order to improve reliability of your service, then my understanding is that it is fine to fulfill the GDPR data deletion requests withing one month period, not immediately.