Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Canon's cloud platform has lost users' files and can't restore them

digitalcameraworld.com

Post reply on HN
Print view View original

331–340 of 419 posts

Re: Canon's cloud platform has lost users' files and can't restore them

#331

Post by _nato_ »

zxcvbn4038 wrote: post #206

This sounds to me like the classic “replication is not backups” situation where (at best) all of the user files were stored in RAID array someplace and that is what the malware ate. If there had been actual backups and effective backups then it should have been trivial to restore non-corrupted files. It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually…

There was an event when a startup I was at asked Basho (they were the company behind Riak db) about backing up our data. Backing was a little side-feature that was possible to rig up, but I recall they looked at this inquiry as if I had two heads -- as if to say, it's replicated, why jump the shark? There was a bug with one of the Riak releases, and all the data was lost. (When we scaled up with this buggy Riak release, the empty node assumed master roll, and all the child nodes went, ah... the new state has no data, let's all delete records 0..k. Fun times.)

Re: Canon's cloud platform has lost users' files and can't restore them

#332

Post by rsync »

sandworm101 wrote: post #279
crazygringo wrote: post #271

Earlier quoted context omitted.

Do any cloud providers create backups on top of replication though? Backing up databases (terabytes) is feasible, they're not that big. But an entire cloud storage for photo and video for millions, we're talking maybe exabytes. The notion of making "separate" backups seems cost-prohibitive. I am curious, though -- for services like Dropbox or Google Drive, how many replicas are there of your files? I know there must…

>> we're talking maybe exabytes I don't see the size as the big problem. If one cloud can handle the size, the backup storage system can too. For me, the real issue is timing. How do you backup a cloud full of constantly changing data? Do you draw a line in the sand, an image of the cloud state at a particular moment? Is that even possible? You have to do backups of smaller chunks, individual accounts, but eventually…

"For me, the real issue is timing. How do you backup a cloud full of constantly changing data? Do you draw a line in the sand, an image of the cloud state at a particular moment?"

I can only describe what we do, and, of course, there is an enormous scale difference here, but ...

Every single rsync.net account is it's own ZFS filesystem which means every single account gets snapshotted[1] nightly on a schedule. This means that the enormous operation of "backing up" all of rsync.net happens in small, manageable, chunks.

Of course, the ZFS snapshots of a customer account are not "backups" per se, but if a customer choose "geo-redundant" storage in another facility, it is those very same snapshots that we (zfs) send over. Those are, indeed, backups.[2]

The most interesting part, in my opinion, is that the daily/weekly/monthly snapshots are immutable. So you can publish your rsync.net credentials or suffer a disgruntled employee or ransomware attack, etc., and those snapshots remain safe - they are read-only.

[1] Accessible, browseable, in ~/.zfs/snapshot

[2] GR storage costs 1.75x normal pricing.

Re: Canon's cloud platform has lost users' files and can't restore them

#333

Post by benatkin »

chrismorgan wrote: post #234
zxcvbn4038 wrote: post #206

This sounds to me like the classic “replication is not backups” situation where (at best) all of the user files were stored in RAID array someplace and that is what the malware ate. If there had been actual backups and effective backups then it should have been trivial to restore non-corrupted files. It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually…

The original story spoke of the rumours of malware involvement, but in the update which forms the first half of the linked article, Canon says there was no malware involved. (“Replication isn’t backup” still applies, of course.)

accidental malware (I get that the name implies malicious and deliberate but that never stops definitions from expanding)

Re: Canon's cloud platform has lost users' files and can't restore them

#334

Post by pravda »

_nato_ wrote: post #331
zxcvbn4038 wrote: post #206

This sounds to me like the classic “replication is not backups” situation where (at best) all of the user files were stored in RAID array someplace and that is what the malware ate. If there had been actual backups and effective backups then it should have been trivial to restore non-corrupted files. It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually…

There was an event when a startup I was at asked Basho (they were the company behind Riak db) about backing up our data. Backing was a little side-feature that was possible to rig up, but I recall they looked at this inquiry as if I had two heads -- as if to say, it's replicated, why jump the shark? There was a bug with one of the Riak releases, and all the data was lost. (When we scaled up with this buggy Riak relea…

BTsync did that to me!

One computer had a hard drive failure, BTSync deleted all the files from the computer that didn't.

Doubleplus ungood.

Re: Canon's cloud platform has lost users' files and can't restore them

#335

Post by jrumbut »

bscphil wrote: post #292
zxcvbn4038 wrote: post #206

This sounds to me like the classic “replication is not backups” situation where (at best) all of the user files were stored in RAID array someplace and that is what the malware ate. If there had been actual backups and effective backups then it should have been trivial to restore non-corrupted files. It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually…

> It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually the one thing they should have backed up because all of the smaller files can be regenerated from the raw ones. Ironically my experience has been exactly the opposite. It's the demosaic-ed, fully developed copies of my photos that are larger and harder to preserve than the original RAWs. And these f…

This, like reproducibility in machine learning workflows, is one of those things that needs to be baked in at the start or will become a complete nightmare.

Re: Canon's cloud platform has lost users' files and can't restore them

#336

Post by foo_barrio »

teawrecks wrote: post #327
bscphil wrote: post #292

Earlier quoted context omitted.

> It also sounds like someone made the decision not to backup the raw images because they were “big” - that is actually the one thing they should have backed up because all of the smaller files can be regenerated from the raw ones. Ironically my experience has been exactly the opposite. It's the demosaic-ed, fully developed copies of my photos that are larger and harder to preserve than the original RAWs. And these f…

I'm not sold. In the cases where you lose the raw vs a post processed image, one of those is significantly more lost information. It may be non trivial to reproduce a post process image from a raw image, but the other way around is usually impossible. In one case you've lost original sensor information, in the other case you've lost some parameters and possibly the algorithm used. The former is infinitely more diffic…

I think that's a really shallow way to look at it. For a professional photographer it's the final version of the file that is actually sold, not the RAW or film negative. These final files are often not some 300x200 web preview. These are full sized images. You can argue the final edit has more value and is the one to preserve. Ansel Adams broke the process down into 3 parts with the initial capture being but the first. So losing the final edits is losing 66% of the total work.

I'm only a hobbyist photog and the bday parties and weddings I've shot result in thousands of images. I can't imagine having to shoot multiple events every week for years. For a pro, the edits represent thousands of hours of work. That the work can be redone theoretically might mean very little to a working photographer.

Re: Canon's cloud platform has lost users' files and can't restore them

#337

Post by aaronmdjones »

soco wrote: post #40
raverbashing wrote: post #33

Earlier quoted context omitted.

You need to denormalize the non-forgettable information. This way you don't have to worry about what you can delete or not. Store Invoices, billing, etc, separately. These also usually have a fixed pre-determined retaining policy time. Past that time? Anonymize/aggregate or just delete it.

Legislation can change and what you once thought to be forgettable becomes suddenly not.

It'd be extremely unusual for such changes to be retrospective. For example, if you are currently required to keep certain kinds of financial records for 3 years, and you delete records older than that, you won't be held at fault when that law gets updated to say 5 years, and you then fail to produce a 4-year-old document; it has not yet been a year since the new law took effect.

This extends to criminal law too, for example with statutes of limitation. If, 20 years and 1 day ago, there was a 20-year statute of limitations for murder charges, you cannot be charged for committing a murder on that day (barring extraneous circumstances, such as tolling); even if a law was passed the day after said murder removing this limitation.

Re: Canon's cloud platform has lost users' files and can't restore them

#338

Post by amiga_500 »

MaxBarraclough wrote: post #286
amiga_500 wrote: post #268

Earlier quoted context omitted.

If you retrieve the data slowly it's cheap. It's expensive if say you are a retailer who needs their database restored asap. I use a Linux perl client!

I suppose the biggest risk is failure-to-pay, as with all cloud backups/storage. If I allow my payment card to expire, Amazon aren't obligated to continue to store my data. If I drop off the grid for an extended holiday, that could be a real risk. To my knowledge, Amazon offer no means of prepaying.

You get emails. Imho it's not a big risk.

Re: Canon's cloud platform has lost users' files and can't restore them

#339

Post by soco »

aaronmdjones wrote: post #337
soco wrote: post #40

Earlier quoted context omitted.

Legislation can change and what you once thought to be forgettable becomes suddenly not.

It'd be extremely unusual for such changes to be retrospective. For example, if you are currently required to keep certain kinds of financial records for 3 years, and you delete records older than that, you won't be held at fault when that law gets updated to say 5 years, and you then fail to produce a 4-year-old document; it has not yet been a year since the new law took effect. This extends to criminal law too, for…

Well, yes, but the topic here was data. And if at some time you thought it was forgettable and you scheduled it for deletion, once the law changes you should make damn sure all code involved in that deletion was checked or the data will be gone - against the new law. A big opportunity for human error here.

Re: Canon's cloud platform has lost users' files and can't restore them

#340

Post by returningfory2 »

xyzzy_plugh wrote: post #308
returningfory2 wrote: post #278

Earlier quoted context omitted.

> Do any cloud providers create backups on top of replication though? Yes - I work at one of the FAANGs for a team that's doing precisely this. We develops an internal disaster recovery tool that creates backups of data files that can't be touched by the creating application, and that can be read back in a disaster event to recover the data.

Are those backups for you or for your customers? The parent is referring to cloud providers, e.g. AWS S3.

both
Post reply on HN