Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Canon's cloud platform has lost users' files and can't restore them

digitalcameraworld.com

Post reply on HN
Print view View original

51–60 of 419 posts

Re: Canon's cloud platform has lost users' files and can't restore them

#51

Post by akerro »

Sharlin wrote: post #28
StreamBright wrote: post #26

Earlier quoted context omitted.

Or maybe an engineering mistake? I do not think you can claim anything without knowing the details. AWS lost customer data too, I would not try to assign blame without reading the post mortem.

According to TFA (which apparently nobody reads) it most certainly was an engineering mistake. Files in long-term storage were accidentally subject to the same code that deletes files in short-term (30 day) storage.

It's sound similar to "junior dev had access to prod database" scenario which really isn't engineering mistake but process/administration.

Re: Canon's cloud platform has lost users' files and can't restore them

#52

Post by Hamuko »

mschuster91 wrote: post #38
klodolph wrote: post #2

Lesson for those designing cold-storage solutions--design your system, as much as reasonable, to not support deletion operations. The reason why files get lost is usually due to software bugs, bad configurations, and operator errors. Design your system to protect against these things.

The problem with that is that under GDPR you have to support full deletion of an user's data.

Not necessarily if you have a valid reason to storing the data. You might for example have to keep user's billing data around for legal reasons.

Re: Canon's cloud platform has lost users' files and can't restore them

#53

Post by ChrisMarshallNY »

henvic wrote: post #36

Everyone should be looking for a backup strategy that involves at least a hot backup and a cold backup, with some sort of offline validation. However this is easier said than done (because now you've to deal with multiple file systems, decisions about cryptography, lack of interest in creating something that you hope you'll never need, and so on). I wish there was a service (and don't tell me about Backblaze because…

I have a multi-dimensional backup strategy, which includes “cloudy” backups of my most important assets, using services like GitHub, local hourly NAS (Synology double-redundant RAID-like) incremental backups of my whole system, and running “hot” backups of my system and development volumes; automatically updated every four hours (or on-demand. It's a bootable CCC clone).

I seldom need to use anything more than my “hot” backups, but have occasionally needed to restore individual files from NAS.

Whether or not I have a backup is never even something I think about at all, which is a big weight off my shoulders.

But photos and videos are a different matter from code (my main assets). They require a lot of space. It’s easy for me to be smug about backups; but photographers have a much more intense set of assets.

Re: Canon's cloud platform has lost users' files and can't restore them

#54

Post by DaiPlusPlus »

dannyw wrote: post #20
SturgeonsLaw wrote: post #7

I take nightly images of my computer's primary drive and replicate them offsite, with alerting if it fails and automated scheduled backup restore testing. It blows my mind to think that some random schmuck like me has better backup procedures than these multibillion dollar global corporations. Companies that don't invest in their IT end up paying for it tenfold down the track.

On topic topic, can I get HN’s advice on my backup strategy? All important files on Synology Ds218+ NAS, which has 2x12TB helium HDDs in mirror. Daily HyperBackup to Google Drive. I test restoring files on a monthly basis. Email alerts on failure. Anything I can improve?

Google, as a company, is not reliable - I refuse to host anything critical on Google Drive (even G Suite) because there's zero recourse or accountability if a bug in Google's software or if Google arbitrarily decides to ban you for life from their services (e.g. due to misidentifying you from a dodgy YouTube user or repeated Google Play Store policy violations) then they will delete all of your personal and business accounts and you're SOL.

What you have is fine for short-term recovery - but I'd make sure you have a long-term and/or cold-storage option set up. It doesn't need to be anything particularly fancy: I'd get a rugged+ portable+external single-drive USB enclosure with a single 16TB drive and have the Synology do a backup of your most critical data onto that drive and store it in your bank's safety deposit box or better yet: leave it with a trusted friend who lives in rural Minnesota or similar.

It goes without saying to encrypt that drive as you won't have full custody of it: but use a simple, proven encryption scheme with a large ecosystem that you know that you'll be able to decrypt in 10-20 years' time.

For backup drives that I keep local to me, I refuse to encrypt them because (in my experience) the possibility of being unable to decrypt data in a desperate or urgent situation just makes me wince.

Re: Canon's cloud platform has lost users' files and can't restore them

#55

Post by saagarjha »

mschuster91 wrote: post #43
saagarjha wrote: post #31

Earlier quoted context omitted.

Oh, really? Can you actually be fined for destroying disks containing customer information?

Yes! Accidental loss is a data breach, see page 7ff of http://ec.europa.eu/newsroom/document.cfm?doc_id=47741

Interesting, they seem to call this an “availability breach”; of course, they require disclosure if it affects people. Do data storage providers need to essentially SLA, then?

Re: Canon's cloud platform has lost users' files and can't restore them

#56

Post by ramraj07 »

mixedbit wrote: post #46
klodolph wrote: post #2

Lesson for those designing cold-storage solutions--design your system, as much as reasonable, to not support deletion operations. The reason why files get lost is usually due to software bugs, bad configurations, and operator errors. Design your system to protect against these things.

Both AWS S3 and Google cloud storage buckets have an option that makes it impossible to delete stored objects for some period of time. The option was added for some legal compliance reasons, but I find it useful as an extra safeguard that important service data is not accidentally or maliciously deleted.

You could always not pay the bill :)

Re: Canon's cloud platform has lost users' files and can't restore them

#57

Post by DaiPlusPlus »

_Microft wrote: post #9
hxegon wrote: post #3

wow this is bad. I know you should keep local backups too but if you shoot RAW then that quickly becomes cumbersome, and only using cloud backup looks very convenient. If I were a professional photographer missing client material or someone who lost potentially irreplaceable memories I'd want a hell of a lot more then just "[Cannon's] deepest apologies".

The software and service might have been ... provided "as is", without warranty of any kind, expressed or implied, including but not limited to...

> provided "as is", without warranty of any kind, expressed or implied, including but not limited to...

IANAL, but when a service is advertised as being ideal for storing your work or photos, isn't that directly implying that there is a warranty that their service is fit-for-purpose? You can't advertise something in big lettering and then countermand that in the small-print - so those magic words certainly don't shield the company from liability at all.

I understand that the "without warranty of any kind, expressed or implied"-line we see in software-licenses and EULAs is when software is distributed without consideration (e.g. open-source software), but when there is consideration (i.e. people paying Canon to host their photos...) then there's a liability if Canon lost peoples' data - so I understand they will be sued for this if anyone lost anything of value. At the very least a 100% refund...

They only way I can see Canon getting out of this is if they had prominent warnings displayed throughout their service's UX advising their users that their service was not suitable for long-term storage of valuable data.

Again, IANAL - can anyone is is a lawyer chip-in?

Re: Canon's cloud platform has lost users' files and can't restore them

#58

Post by Hamuko »

CalRobert wrote: post #5
klodolph wrote: post #2

Lesson for those designing cold-storage solutions--design your system, as much as reasonable, to not support deletion operations. The reason why files get lost is usually due to software bugs, bad configurations, and operator errors. Design your system to protect against these things.

If memory serves, a class of Glacier storage (AWS) isn't practical to use in the EU because the inability to delete makes it impossible to serve "Right to Erasure" GDPR requests.

Not necessarily. While it's not set in stone, GDPR right to erasure might not be in conflict with backup retention.

>The GDPR is open to interpretation, so we asked an EU Member State supervisory authority (CNIL in France) for clarification. CNIL confirmed that you’ll have one month to answer to a removal request, and that you don’t need to delete a backup set in order to remove an individual from it. Organizations will have to clearly explain to the data subject (using clear and plain language) that his or her personal data has been removed from production systems, but a backup copy may remain, but will expire after a certain amount of time (indicate the retention time in your communication with the data subject). Backups should only be used for restoring a technical environment, and data subject personal data should not be processed again after restore (and deleted again). While this adds some complexity, it allows organizations to have some time to re-engineer their data protection processes.

https://blog.quantum.com/2018/01/26/backup-administrators-th...

Re: Canon's cloud platform has lost users' files and can't restore them

#59

Post by chrismatheson »

mixedbit wrote: post #46
klodolph wrote: post #2

Lesson for those designing cold-storage solutions--design your system, as much as reasonable, to not support deletion operations. The reason why files get lost is usually due to software bugs, bad configurations, and operator errors. Design your system to protect against these things.

Both AWS S3 and Google cloud storage buckets have an option that makes it impossible to delete stored objects for some period of time. The option was added for some legal compliance reasons, but I find it useful as an extra safeguard that important service data is not accidentally or maliciously deleted.

I suppose that feature would be incompatible with user data in a GDPR world ??

Re: Canon's cloud platform has lost users' files and can't restore them

#60

Post by DaiPlusPlus »

ponker wrote: post #12

The camera companies are so bad at software that the software companies are building better cameras than them... in the few millimeters thickness of a phone! I salivate for the day that Google or Apple buys Nikon or Leica or whatever and shoves all of that computational photography goodness under a 24x36mm camera sensor.

As of 2019, Apple sources their iPhone camera sensor components from Sony and the camera lenses from Largan Precision (and other companies). The software that Apple contributes is important - yes (after-all, this is software-defined-photography now, with Portrait Mode and that fancy but fake depth-of-field stuff in the single-lens iPhones) - but let's not pretend Apple is "a camera company".
Post reply on HN