Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Canon's cloud platform has lost users' files and can't restore them

digitalcameraworld.com

Post reply on HN
Print view View original

251–260 of 419 posts

Re: Canon's cloud platform has lost users' files and can't restore them

#251

Post by JoeAltmaier »

F_J_H wrote: post #248
wodenokoto wrote: post #156

Earlier quoted context omitted.

No, that comment sounds like the insurance job described in Fight Club[1] > Narrator: A new car built by my company leaves somewhere traveling at 60 mph. The rear differential locks up. The car crashes and burns with everyone trapped inside. Now, should we initiate a recall? Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A…

A farmer's donkey dies. His son gets an idea on how to make some money. Sometime later he turns up at dinner with $200. "Where did you get that money?" His father asks. "I sold raffle tickets for the donkey" his son replied. "What?!?" Shouts his father. "You can't sell raffle tickets for a donkey that you know has already died! Weren't people mad?" "Just the guy who won" said the son, "so I gave him his money back."

Really, he'd want the value of a donkey - not his money back.

But with a raffle, the 'take' can easily exceed the cost of the prizes. So I guess the son would still make out ok.

Re: Canon's cloud platform has lost users' files and can't restore them

#252

Post by gameswithgo »

ibeckermayer wrote: post #226
zxcvbn4038 wrote: post #206

This sounds to me like the classic “replication is not backups” situation where (at best) all of the user files were stored in RAID array someplace and that is what the malware ate. If there had been actual backups and effective backups then it should have been trivial to restore non-corrupted files. It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually…

What’s the difference between replication and backups? Is the distinction that backups must be stored on separate infrastructure, whereas replication might still be 1 or 2 points of failure?

A backup would be resistant to someone accidentally deleting all the files, for instance. Replication would not.

Re: Canon's cloud platform has lost users' files and can't restore them

#253

Post by virgilp »

nix23 wrote: post #105
virgilp wrote: post #94

Earlier quoted context omitted.

They obviously do have _some_ backups (since were able to restore all videos and all photos - but not all photos at the original resolution). In the end it's all a tradeoff... ignoring cost, we all know what to do to get pretty safe storage. But how much are you really willing to pay, per month, for that safety?

>but not all photos at the original resolution Thats why you backup the original data and NOT the downscaled etc one. >But how much are you really willing to pay, per month, for that safety? You have to backup ALL your customers data, otherwise DONT go into file-hosting business.

But they did backup the original data - the article states that the problem was with deploying wrong cleanup code on the long-term storage (that was supposed to be only deployed for the short-term storage). How do you guard against that? You could store everything in backup storage & never delete it, but that has implications:

- privacy

- legal (GDPR)

- cost etc.

It's basically a "black swan" event, they designed their system without anticipating this particular failure mode. It's easy to claim in hindsight that "they should've or they should not go into that business" but seriously.... look in the mirror. Are you as tough on yourself when it comes to your customers? Can you honestly claim that there's absolutely no failure mode that could result in customer data loss?

Re: Canon's cloud platform has lost users' files and can't restore them

#254

Post by ashtonkem »

mumblemumble wrote: post #243
lukevp wrote: post #232

Earlier quoted context omitted.

Replication is about having data level redundancy to protect from drive failure. Backups are about having point in time snapshots of the system state, and about having them tiered from a location perspective. The 3-2-1[1] principle says to have 3 total copies, 2 of which are local but on different devices, and 1 of which is offsite. This gives you tiers of recoverability. It’s important from a backup perspective that…

I would add to that, that the offsite one should also be off line .

Or at least in an append only data store.

Re: Canon's cloud platform has lost users' files and can't restore them

#255

Post by imglorp »

throwaway0a5e wrote: post #231
imglorp wrote: post #194

Earlier quoted context omitted.

AND in the Suzuki Samuri rollover deaths (I knew one victim): where they knew there was an issue very early and had a "crisis plan". https://en.wikipedia.org/wiki/Suzuki_Motor_Corp._v._Consumer...

And meanwhile something like 20% of Bronco IIs got rolled and nobody cared. But as soon as someone puts an Explorer on it's roof or a Samurai barrel rolls through an intersection and there's a problem. Pretty much that whole generation of midsize SUVs rolled easily. That's just one of the engineering trade-offs all those vehicles made and anyone can tell you that. If Consumer Reports was picking my SUV as the one to…

I think GP's point was that they did the fight club math ahead of time, knowing they'd have a certain number of deaths and lawsuits, and proceeded to market with that plan.

Spin/media/marketing/CR/NHTSA after it's all public is a bigger subject, yes.

Re: Canon's cloud platform has lost users' files and can't restore them

#256

Post by throwaway0a5e »

droopyEyelids wrote: post #220
throwaway0a5e wrote: post #180

Earlier quoted context omitted.

>Intent tends to matter with the law. Ford knew there was a problem, knew they could fix it, and elected not to because "it's a small number of deaths". You can replace things, not people. Per the link posted by the GP the Pinto did not have any problems relative to other cars of the same type and era. Ford's "intent" (if you can call it that since they didn't know the future at the time) was to say "err, we don't re…

Seems like you're failing to address the 'knew they were going to kill people' part. Everyone can always do better, but everyone doesn't know exactly which of their decisions will end up killing another person. If you DO know that, then you're obligated to change things.

>Seems like you're failing to address the 'knew they were going to kill people' part.

Damn near every engineering decision could lead to some sequence of events that kills someone and given infinite time will eventually do so. This is a core concept of pretty much every ethics in engineering class that many people here have taken in order to obtain their degree.

Don't slap doors befitting a missile silo on your car? If your cars rick up enough miles someone's gonna die in a T-bone eventually.

Build an overpass with small beams and a support on the median instead of big expensive beams and no support? Someone's gonna hit that support eventually.

The sorts of excesses that can prevent unlikely deaths are justifiable in individual and in the abstract but at a societal level we cannot afford to think this way. As we get richer we can afford to hedge against more and more remote failure modes but even then at some point the risk is low enough.

In the 1950s we couldn't justify seat-belts in all cars. In the 1980s we couldn't justify making cars as rigid as we do today. Between wealth and technological progress things are generally getting better all the time but to hold the past to the standards of the present is asinine.

Re: Canon's cloud platform has lost users' files and can't restore them

#257

Post by hvidgaard »

qntmfred wrote: post #200
hvidgaard wrote: post #73

Earlier quoted context omitted.

This is not one of them. Ensuring proper data loss protection is something both Amazon and Google are very good at. A company of Canons size could easily get the proper guidance to avoid it. But it would not have been as cheap as storing the files on your own servers. So this happened because Canon did not want to spend the money on doing it right with offline backups or append only storage in a cloud with enough nin…

Companies of Canon's size can be constrained for resources just as much as any other company. Canon is largely a legacy company at this point, with significantly declining revenue the last several years. https://i.redd.it/lrlfen2x7sm31.jpg

Even more reason to spend a little bit more to make this right. It's not expensive to get a proper expert on the matter to figure things out. The last thing they want is bad publicity for things that could earn them recurring revenue.

Re: Canon's cloud platform has lost users' files and can't restore them

#258

Post by microcolonel »

richardwhiuk wrote: post #120
microcolonel wrote: post #99

Earlier quoted context omitted.

Filtering this stuff from a master transaction log is kinda ludicrous, not looking forward to implementing that.

I don't think the GDPR imposes a requirement to actively filter backups.

It's not a backup, it's the authoritative dataset.

Re: Canon's cloud platform has lost users' files and can't restore them

#259

Post by richardwhiuk »

DaiPlusPlus wrote: post #60
ponker wrote: post #12

The camera companies are so bad at software that the software companies are building better cameras than them... in the few millimeters thickness of a phone! I salivate for the day that Google or Apple buys Nikon or Leica or whatever and shoves all of that computational photography goodness under a 24x36mm camera sensor.

As of 2019, Apple sources their iPhone camera sensor components from Sony and the camera lenses from Largan Precision (and other companies). The software that Apple contributes is important - yes (after-all, this is software-defined-photography now, with Portrait Mode and that fancy but fake depth-of-field stuff in the single-lens iPhones) - but let's not pretend Apple is "a camera company".

I expect Apple sold more camera than Canon last year.

Re: Canon's cloud platform has lost users' files and can't restore them

#260

Post by IshKebab »

klodolph wrote: post #2

Lesson for those designing cold-storage solutions--design your system, as much as reasonable, to not support deletion operations. The reason why files get lost is usually due to software bugs, bad configurations, and operator errors. Design your system to protect against these things.

Good luck complying with the GDPA.
Post reply on HN