Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DigitalOcean lost our data and gave us $500

dfernandez.me

Post reply on HN
Print view View original

51–60 of 69 posts

Re: DigitalOcean lost our data and gave us $500

#51

Post by gregd »

astrodust wrote: post #37
gregd wrote: post #18

This is 2013. Why are we still talking about backups as a lesson learned? Is it because startups are skimping on Sys Admins?

It's because some startups have developers that open w3schools, start typing examples, and somehow ship a quasi-working proof-of-concept that goes into production. There's a bit of "if it ain't broke don't fix it" here, but a whole lot of "get with the program" still required.

Well as a professional Systems Administrator, it pisses me off more than it probably should. It's like you want to know why I'm worth what I'm asking because when your shit falls down and goes boom, I'll get you back up and operational in minutes or an hour.

Because it's my fucking job to help you manage your IT risks. Azure, Heroku, AWS aren't replacements for Systems Administration, they're just tools in my arsenal. I don't understand the mentality it takes to go into business (beta or not) without having SOME understanding of your risk. The fact that DO paid you a not insignificant amount due to downtime, means you're damn lucky.

Re: DigitalOcean lost our data and gave us $500

#52

Post by rb2e »

The $500 credit from DO is quite reassuring. Usually if the HD fails and your data is lost, your out of luck. I hear the "horror" stories of some hosts reusing consumer Hard Drivers between servers so learned, Your data is your responsibility. I'm glad the OP had backups but these failures happen, thankfully DO had the business sense to compensate them.

Seems good advertising for DO, as any knowledgable system admin knows Drives fail. DO could have not done anything.

Re: DigitalOcean lost our data and gave us $500

#53

Post by fiatmoney »

KaiserPro wrote: post #47

This might sound a bit glib, but raid 5 shouldn't really be used in modern storage. If you ignore the performance issues (which can vary by device) its just not safe. Depending on the size of drive can take anywhere up 30hours+ to rebuild. bear in mind that you tend to use disks that are all the same batch, it leaves you in the danger zone for far too long. Your options are: somesort of clever RAID (ZFS type thing) A…

For SSDs, where the time-to-read/write-full-capacity is typically much less than HDDs (both due to higher speed & lower capacity), it can be less of a poor decision. SSDs also have somewhat more advanced machinery for data integrity checking and slightly friendlier failure modes (e.g., the sectors "wear out" over time, but the firmware tends to warn you as that starts to happen, and you're not going to hit a sudden mechanical failure).

Re: DigitalOcean lost our data and gave us $500

#54

Post by kylec »

DigitalOcean's pricing page indicates that "All cloud hosting plans include automated backups". (https://www.digitalocean.com/pricing) From the email you received, it sounds like this is clearly not the case. I wonder what other claims DigitalOcean is making that are not true.

Re: DigitalOcean lost our data and gave us $500

#56

Post by AYBABTME »

level09 wrote: post #29

The author is sweet, his conclusion was "always backup your data" if it was me I would probably say "I'm moving away, will never trust them again on my data" ..

Hardware fails, there's no way around it.

At 5$/month, I think it's not too much an investment to have some basic redundancy if you care about your data. Anyways, if your data matters to you, do backups.

Re: DigitalOcean lost our data and gave us $500

#57

Post by ChuckMcM »

dangrossman wrote: post #40
chc wrote: post #12

The abrasive headline is kind of unfortunate, as the actual moral of the story given at the end is exactly the right takeaway: Never assume your hardware is infallible, so always have backups that you know you can use when your server experiences a wildly improbable catastrophe. Also, very impressed by Digital Ocean's response here. Given their reputation as a budget host, they really do put a lot of effort into serv…

> wildly improbable catastrophe Or an extremely probable one like a hard disk failure. They only last a few years; most data centers see an annual replacement rate in the 2-13% range. The failure rate is a known quantity, and their limited 1-3 year warranties that reflect that expectation. There isn't a host I've used more than a few years where I haven't seen hard drives (and power supplies) fail. I don't know if my…

Was going to say the same thing, Dual drive failure on a RAID5 system with five 2TB drives is 1 in 12. With 3TB drives that goes up to 1 in 7.

The underlying issue is that the uncorrectable read error rate is 1 in 10^15 bits, this is just physics (thermal noise, read/write signal loss, etc) But with 8b/10b encoding that is only 90TB worth of bits. Rebuilding a RAID group of 5 with four 2TB "good" drives (8TB of data to be read) you will see a failure in one of the other 4 drives 1 in 11.25 times. (90/8). With 3TB drives 1 in 7.25 times. Using simple mirroring you won't be able to re-silver a mirror 1 in 1:45 or slightly more than 2% of the time for 2TB drives.

Dual parity, or triple mirrors (x3) are now the minimum bars for making storage reliable.

Re: DigitalOcean lost our data and gave us $500

#58

Post by wiredfool »

sillysaurus2 wrote: post #31
wiredfool wrote: post #21

Earlier quoted context omitted.

As long as you issue the pg_start_backup/pg_stop_backup pair and keep the WAL logs. If you don't, then you've got a corrupt backup. At least you would catch that problem in your first test restore.

... what? The point of consistency is that if the power to your server is cut, then you can reboot and pick up precisely where you left off. That means the database on disk must have consistency. Meaning you should be able to copy it at any point in time without any problems. If you can't, then that's not consistency, and if postgres really works that way, then it's failing one of the basic tenants of being a databas…

Re: VMs

Leaving aside the management issues of huge vm images and the less than ideal io performance, the ACID guarantees of pg rely on the underlying hardware obeying some specific restrictions, including real fsync and not lying about when things are on permanent storage. Getting the drives and raid controllers to obey that has historically been a difficult, ongoing job that has to be redone with each new generation of hardware. SSDs have been particularly interesting with that, the actual flush to disk can be quite delayed from the logical write. Some have supercaps, some don't. Those that don't are vulnerable to power losses while the data is still in the drive's ram awaiting a block erase and write. The IDE drivers used to flat out lie. Enterprise SAS drives often come with the write caching turned on (since it looks better in benchmarks) even though they're often times used behind a battery backed raid controller.

Adding a VM layer to that just to get snapshots seems overly complicated and prone to issues.

Re: DigitalOcean lost our data and gave us $500

#60

Post by richardjs »

kylec wrote: post #54

DigitalOcean's pricing page indicates that "All cloud hosting plans include automated backups". ( https://www.digitalocean.com/pricing ) From the email you received, it sounds like this is clearly not the case. I wonder what other claims DigitalOcean is making that are not true.

There is an automated backup system that you have to enable for a droplet, that creates a snapshot every few days. It's a clear part of a droplet's control panel. They began charging for it in July 2013. The price is 20% of the droplet's monthly cost. Sounds like they need to update their pricing page.

This is pertaining to a droplet feature though, and not some low-level backup system. Meaning, it's not as if they're lying about the infrastructure below what a normal customer can see. They just have an erroneous pricing page.

Post reply on HN