Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Our Server’s Hard Drive is Dead. We didn’t have a backup.

blog.method.ac

Post reply on HN
Print view View original

51–60 of 88 posts

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#51

Post by Udo »

Sanddancer wrote: post #37
ashray wrote: post #9

I don't understand, you had a backup HD - that means you had a RAID setup. Why didn't your host replace the damaged hard disk ? In my experience hosts usually monitor RAID health on their servers and if there is a problem they replace the bad hard drives at the quickest opportunity.. and I'm talking about budget hosts. EDIT: Too many to respond to below so just editing in here. The author mentioned that the primary h…

Repeat after me: RAID is not a backup. It will mitigate certain drive failures, but is not a backup. Period, end of statement. Controllers will forget their info, OSes will eat their partition tables, and will otherwise ruin your data. If you're not backing up your stuff, to a completely separate system, preferably to a completely separate service, you will lose data. Period.

I don't think that's what ashray meant. A backup (as in failover) HD, not an HD that stores backups... Because apparently to add insult to injury, in this case they ignored a failed RAID drive and didn't have backups.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#52

Post by McGlockenshire »

JPKab wrote: post #15

It's noble of you to come clean and own your mistake, but let me say this over and over: You should never, ever provide an environment that stores people's hard work without having professionals who know how to safeguard it. If it makes you feel any better, I recently had to clean up a mess in a huge enterprise IT shop, (if I were to name the organization you would immediately know them) involving hundreds of thousan…

> You should never, ever provide an environment that stores people's hard work without having professionals who know how to safeguard it.

Even if they do know how to safeguard the data, that doesn't mean that everything else is going to work properly.

I had recently taken over IT after working for six years as a developer. In fact, this happened only a month or so in to my new role.

Our mail server died. Three out of four drives in the hardware RAID 10 failed. I'd been seeing bounces to root@localhost from root@localhost in the nightly reports, but the way things were configured made it nearly impossible to figure out where the mails were coming from. Thanks, Zimbra. We speculate that these were constant alerts from our RAID card notifying us of the impending disaster.

Oh, and the only backups for the mail store were on the machine itself, and in the local Thunderbird installs that half the company used instead of the Zimbra web interface. The machine was in a colo downtown, not local, and running backups over our pathetic little DSL connection was unmanageable.

Both of these things were known problems, both marked high priority, but both months away from being addressed when things went south.

This happened on a Friday. By Monday morning, I'd moved us over to a hosted service, manually sorted all of the mail that hit a catch-all mailbox on a VM I'd set up. By Tuesday, I'd audited every one of our other machines to make sure that mail to root was deliverable (it wasn't in about a dozen machines) and that every machine with hardware RAID had both local and remote monitoring.

Some people, including Directors and C-levels, lost up to ten years of mail. It was the worst IT disaster the company ever faced. But that's not the worst part. No, the worst part is that we're in the IT industry, and knew the entire time that what we were doing was wrong... fixing it had just never been prioritized before, because it wasn't seen as super urgent that it be fixed.

That lesson has been learned.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#53

Post by Gygash »

ashray wrote: post #11
gregd wrote: post #8

Earlier quoted context omitted.

I mean that Systems Administration is as thoroughly exhausting a career path as programming. They didn't lose information because they coded something wrong or inadequately.

The most basic backup would be to have a remote backup of your files and databases. That's a simple shell script and cron job combination. I can understand that one may not get around to setting it up (I had a massive project and only started doing this after about 7 years of running it..) but it's definitely very important and doesn't need a ton of Sysadmin experience or setup time.

> a remote backup of your files and databases

Are you guaranteeing that those files are in a good state when you ship them off? (i.e., flushed and synced to disk)

How about the DBs? Are you backing up the *SQL data files directly (and if so, are you guaranteeing that _those_ are in a good state?), or doing an SQL or other export?

Exactly which files do you need to backup -- user generated content, system configuration, logs, spool files? Do those files give you enough information to rebuild your machine(s) from bare metal? Or have you have documented your setup procedures?

Where are you storing these backups? And how many copies of them, and on what frequency, should you be keeping?

Are you testing your backups? Are you testing your recovery procedures? If not, how do you _know_ they're all working?

---

Sorry for the question dump, but there is a fair bit more to even basic backups than just syncing your filesystem to S3 / tarsnap / The Cloud TM every day.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#54

Post by nwilkens »

Brandon0 wrote: post #29
nwilkens wrote: post #18

I see this too many times.. and have read about this more than once on HN in recent memory. Hire a proper system administration company early to work with you on these types of things. There are many companies out there that do this. I happen to run a company that does this, so I know that you can add an expert admin to your team for $100-200/mo.

That is actually surprisingly cheap. Care if I ask what types of services one would get at those rates? You're absolutely right though, for a company like OP's, if they are so short on cash, it makes a lot of sense to get someone in even if just for the week to address these types of fundamental problems.

For a monthly service, you generally receive an initial: - System architecture review - Backup strategy / DR review - Security scan, and detailed review - System monitoring design, and implementation

and on-going: - 24x7 monitoring, and response to outages - Server patch management - Ad-hoc system admin time available to be used on-demand

Many more details, and capabilities, but you get the idea ;)

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#55

Post by philfreo »

jefe78 wrote: post #26
gregd wrote: post #8

Earlier quoted context omitted.

I mean that Systems Administration is as thoroughly exhausting a career path as programming. They didn't lose information because they coded something wrong or inadequately.

Perhaps it's me, but I don't see the roles of SA and Dev as interchangeable. Yet, HN seems to present a world where all Devs are SAs. As a SA, it's frustrating. All of that to say; I agree with what you're saying. I just wonder how to get the message across that maybe both roles aren't the same...

This is likely because many of us are part of small startups where the roles are often interchangeable in the early days: the same small engineering team is responsible for both "writing code", backups, infrastructure, and everything else technical.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#57

Post by [deleted] »

gregd wrote: post #16
jrussbowman wrote: post #13

Earlier quoted context omitted.

glad I wasn't the only one who had that as a first thought. as an ops guys I wasn't sure if I should be offended or just shake my head at the irony of it.

Well apparently someone didn't like my comment since it got down-voted. Whatever. As someone with a background in systems administration, it bothers me when this profession gets left out of the equation, far too often.

[deleted]

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#58

Post by ashray »

Gygash wrote: post #53
ashray wrote: post #11

Earlier quoted context omitted.

The most basic backup would be to have a remote backup of your files and databases. That's a simple shell script and cron job combination. I can understand that one may not get around to setting it up (I had a massive project and only started doing this after about 7 years of running it..) but it's definitely very important and doesn't need a ton of Sysadmin experience or setup time.

> a remote backup of your files and databases Are you guaranteeing that those files are in a good state when you ship them off? (i.e., flushed and synced to disk) How about the DBs? Are you backing up the *SQL data files directly (and if so, are you guaranteeing that _those_ are in a good state?), or doing an SQL or other export? Exactly which files do you need to backup -- user generated content, system configuratio…

Files are pretty straightforward, especially in a possibly low volume - startup that cannot afford a dedicated sysadmin stage, not going to address that here.

Simple database backup would be to replicate to a slave (just addressing mysql here for a simple case). Stop replication, backup, restart replication. Easy as that. More complicated scenarios need more complicated setups but hey, we're not talking about saving the last 0.0000001 seconds of data here, this is about losing all your data, from day 1.

Obviously I cannot comment on the specific backup needs for these guys since I don't know their app. However, I can still ensure that an app that is running on a shared hosting environment (meaning - it's not that complex..) has a reasonably good (24 hour snapshot ?) backup potential within a few minutes.

I think it'd have been a decent compromise if they even had a backup from yesterday with a few minutes/hours of effort upfront.

Not saying here that a sysadmin role isn't justified, just that there are a few steps you can follow so that while you don't have a sysadmin, nothing gruesome happens to you.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#59

Post by nwilkens »

rorrr2 wrote: post #50
nwilkens wrote: post #18

I see this too many times.. and have read about this more than once on HN in recent memory. Hire a proper system administration company early to work with you on these types of things. There are many companies out there that do this. I happen to run a company that does this, so I know that you can add an expert admin to your team for $100-200/mo.

> you can add an expert admin to your team for $100-200/mo Hahaha.

It's what we do -- so I laugh with you ;)
Post reply on HN