Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Canon's cloud platform has lost users' files and can't restore them

digitalcameraworld.com

Post reply on HN
Print view View original

311–320 of 419 posts

Re: Canon's cloud platform has lost users' files and can't restore them

#311

Post by zhengyi13 »

site-packages1 wrote: post #295
skoskie wrote: post #270

Earlier quoted context omitted.

That’s a lot harder to pull off. What methods do you use to accomplish this?

Where I worked, weekly point in time backups were required. Those backups were put onto tape drives, those tape drives were set on a pallet and then driven by truck to an offline second location. IMO _that’s_ how it’s done properly.

When we used Iron Mountain at my last job, there were two interesting additional concerns:

1) That nobody in our company actually knew where they were physically stored (insider risk?), but

2) That we had assurance that the physical storage was far enough away that a physical disaster in our area wouldn't also touch where the offline storage was located.

There's a concomitant jump in RTO if that's what you do this, but hopefully that's well understood among the stakeholders.

Re: Canon's cloud platform has lost users' files and can't restore them

#312

Post by tjohns »

crazygringo wrote: post #271
zxcvbn4038 wrote: post #206

This sounds to me like the classic “replication is not backups” situation where (at best) all of the user files were stored in RAID array someplace and that is what the malware ate. If there had been actual backups and effective backups then it should have been trivial to restore non-corrupted files. It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually…

Do any cloud providers create backups on top of replication though? Backing up databases (terabytes) is feasible, they're not that big. But an entire cloud storage for photo and video for millions, we're talking maybe exabytes. The notion of making "separate" backups seems cost-prohibitive. I am curious, though -- for services like Dropbox or Google Drive, how many replicas are there of your files? I know there must…

Google talks a bit about their general backup strategy in the SRE book:

https://landing.google.com/sre/sre-book/chapters/data-integr...

The short version is that the backups are a mix of short-term local backups + backups across the network on distributed filesystems + offline tape backup.

I can't speak to Drive, but the tape backup is certainly used for GMail. (There's a case study about Gmail having to restore from backup in the above link.)

Re: Canon's cloud platform has lost users' files and can't restore them

#313

Post by munk-a »

JulianWasTaken wrote: post #307
kevstev wrote: post #282

Earlier quoted context omitted.

Just using AWS S3 as an example, you can copy a bucket to another bucket (preferably in a different region) with a policy of retaining all versions- preventing the problem of a delete whether malicious or accidental, from being reflected in the backup copy. There is also the option of using Glacier for lower cost long term storage. If you are asking does regular S3 provide a true backup capability out of the box, it…

We don't think AWS does periodic offline backups of stuff stored in S3 so that they don't find themselves in this exact embarrassing scenario? Regardless of whether it's user-facing for AWS users, I'd hope they do, or certainly that they did before they got a good sense of the long-term reliability of S3 as a big system.

I think they do all sorts of backup stuff - and as a consumer of S3 you need to evaluate how resilient their backing up is. S3 is pretty murky about permanence guarantees so I'd always look at making sure there is a separately maintained replication script to some medium I have control over if that data is irreplaceable and the costs of losing it are significant to the business.

Judging those costs and making that call is a complex matter of course.

Re: Canon's cloud platform has lost users' files and can't restore them

#314

Post by dec0dedab0de »

ibeckermayer wrote: post #226
zxcvbn4038 wrote: post #206

This sounds to me like the classic “replication is not backups” situation where (at best) all of the user files were stored in RAID array someplace and that is what the malware ate. If there had been actual backups and effective backups then it should have been trivial to restore non-corrupted files. It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually…

What’s the difference between replication and backups? Is the distinction that backups must be stored on separate infrastructure, whereas replication might still be 1 or 2 points of failure?

I see a bunch of explanations, but I don't think any of them really drove home the reason for why you usually need both.

Imagine if you had a document that stored useful information. You had this document automatically replicated to another system in a different time zone every time there was a change.

You think you're doing great, if there is an outage in the one system you just get your important document from the other system.

Then one day someone accidentally copies and pastes the wrong data into the file. Now what do you do? If you goto your replicated copy it also has the bad data.

The answer is you should have had backups too. so you could go back an hour,a day or a week or even much longer. Depending on how much data you have, how often it changes, how important it is, and how quickly someone would notice bad data.

Re: Canon's cloud platform has lost users' files and can't restore them

#315

Post by hughes »

dharma1 wrote: post #297

Japanese camera companies are really bad at cloud software - Sony Playmemories probably the worst offender. Also the in-camera "mini-apps" tend to be terrible. They make good hardware but horrible software. They should really buy a computational photography company that also knows how to make good web and native apps, like https://skylum.com/

I remember paying $10 to buy a mini-app that added what should have been native functionality to my Sony camera. Then a few months later a firmware update made it incompatible.

Re: Canon's cloud platform has lost users' files and can't restore them

#316

Post by dec0dedab0de »

crazygringo wrote: post #271
zxcvbn4038 wrote: post #206

This sounds to me like the classic “replication is not backups” situation where (at best) all of the user files were stored in RAID array someplace and that is what the malware ate. If there had been actual backups and effective backups then it should have been trivial to restore non-corrupted files. It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually…

Do any cloud providers create backups on top of replication though? Backing up databases (terabytes) is feasible, they're not that big. But an entire cloud storage for photo and video for millions, we're talking maybe exabytes. The notion of making "separate" backups seems cost-prohibitive. I am curious, though -- for services like Dropbox or Google Drive, how many replicas are there of your files? I know there must…

I haven't looked at dropbox in a while, but they at least used to have an option for keeping revisions. It did come with an extra cost.

Re: Canon's cloud platform has lost users' files and can't restore them

#317

Post by abnercoimbre »

gvjddbnvdrbv wrote: post #266
JoeAltmaier wrote: post #251

Earlier quoted context omitted.

Really, he'd want the value of a donkey - not his money back. But with a raffle, the 'take' can easily exceed the cost of the prizes. So I guess the son would still make out ok.

He would want the value of the donkey. The son would offer the value of the tickets. Lawyers would make it somewhere in the middle. Then the lawyers would walk off with all the money.

This has been a valuable thought experiment for me.

Re: Canon's cloud platform has lost users' files and can't restore them

#318

Post by fokinsean »

dannyw wrote: post #20
SturgeonsLaw wrote: post #7

I take nightly images of my computer's primary drive and replicate them offsite, with alerting if it fails and automated scheduled backup restore testing. It blows my mind to think that some random schmuck like me has better backup procedures than these multibillion dollar global corporations. Companies that don't invest in their IT end up paying for it tenfold down the track.

On topic topic, can I get HN’s advice on my backup strategy? All important files on Synology Ds218+ NAS, which has 2x12TB helium HDDs in mirror. Daily HyperBackup to Google Drive. I test restoring files on a monthly basis. Email alerts on failure. Anything I can improve?

Synology just opened a US datacenter for their C2 backup service, it's quite easy to use and IMO better than hosting on Google.

Re: Canon's cloud platform has lost users' files and can't restore them

#319

Post by ISL »

bscphil wrote: post #292
zxcvbn4038 wrote: post #206

This sounds to me like the classic “replication is not backups” situation where (at best) all of the user files were stored in RAID array someplace and that is what the malware ate. If there had been actual backups and effective backups then it should have been trivial to restore non-corrupted files. It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually…

> It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually the one thing they should have backed up because all of the smaller files can be regenerated from the raw ones. Ironically my experience has been exactly the opposite. It's the demosaic-ed, fully developed copies of my photos that are larger and harder to preserve than the original RAWs. And these f…

So far, I have found that Darktable's edits have survived changes in versions. Someday there will probably be breaking changes, but I haven't yet felt it personally.

Re: Canon's cloud platform has lost users' files and can't restore them

#320

Post by jeffbee »

crazygringo wrote: post #271
zxcvbn4038 wrote: post #206

This sounds to me like the classic “replication is not backups” situation where (at best) all of the user files were stored in RAID array someplace and that is what the malware ate. If there had been actual backups and effective backups then it should have been trivial to restore non-corrupted files. It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually…

Do any cloud providers create backups on top of replication though? Backing up databases (terabytes) is feasible, they're not that big. But an entire cloud storage for photo and video for millions, we're talking maybe exabytes. The notion of making "separate" backups seems cost-prohibitive. I am curious, though -- for services like Dropbox or Google Drive, how many replicas are there of your files? I know there must…

A thing stored in google cloud is generally erasure coded, which provides redundancy against the failure of individual devices or hosts, and another copy is stored separately in a geographically separate place. So you might think of it as there being 1.7 copies of your file in each of at least two places.
Post reply on HN