Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DigitalOcean lost our server

murze.be

Post reply on HN
Print view View original

111–120 of 219 posts

Re: DigitalOcean lost our server

#111

Post by xytop »

therealmarv wrote: post #92

No surprise for me. It also happened on DO for me one time. It seemed I've had my machine on a wrong rack/server there. At the end you get what you pay for I would say... you can have luck on your DO server but there is a reason why Rackspace and Amazon still exist nowadays! If you can afford one of the bigger ones go for it.

Rackspace!!1 ha ha ha ha ha.. (neurotic laugh..) They took our whole infrastructure down for a day just because their highly paid Linux support team changed all root passwords and screwed firewalls and added unneed balancers so that servers weren't able to communicate with each other.. We didn't even request anything (as I remember).. Just bought their support and this shit happened. That was a nightmare

Re: DigitalOcean lost our server

#112

Post by ryanmarr »

The comment thread on this post makes me very happy. Having recently switched all my production instances to individual docker containers, I'm very happy to hear that the consensus seems to be that server instances should be killed and spun back up like bacteria. There's hope for humanity yet.

Re: DigitalOcean lost our server

#113

Post by mappu »

vardump wrote: post #103
hyperknot wrote: post #97

Earlier quoted context omitted.

Not a huge company like DO, but iwStack [1] provides a SAN backed cloud, with selectable KVM/XEN instances, custom ISO and virtual network support. The prices are similar to DO. [1] http://iwstack.com/

That's pretty damn interesting for more traditional hosting type usage. Seems like they sell 50 GB of real SAN backed storage for just 3.6€ per month. How can they afford to be so cheap?

That's not even particularly cheap per-GB - BuyVM ( http://buyvm.net/storage-vps/ ) will sell you 250GB for 7USD/mo.

Disclaimer: happy customer

Re: DigitalOcean lost our server

#114

Post by lobster_johnson »

ljoshua wrote: post #101

I hadn't heard of the backup providers mentioned in the article. Does anyone have experience with those or with other recommended solutions? I can re-implement the infrastructure of my VMs fairly easily (thank you Ansible), but backing up content outside of the provider's built-in options is something I haven't played with yet, and would obviously be the crucial piece.

We use Backupninja to drive backups, mostly Duplicity against S3. Some things, like Postgres dumps, are custom commands that just upload individual files to S3. The reason you want to wrap things in Backupninja is to centralize logging, scheduling and monitoring.

Note that we only back up specific things like databases, logs (central syslog server) and data directories. We use Puppet to configure our boxes, and consider everything to be expendable that isn't data.

Re: DigitalOcean lost our server

#115

Post by krinchan »

I use DO as a test bed for a lot of stuff. I ended up destroying/recreating so many droplets I just started using Chef. Also, I come from a major background in AWS, ChaosMonkey in prod, and a very strict push-button + nuke & pave deployment strategy.

So, all this strikes me as "lol you didn't know this?"

Re: DigitalOcean lost our server

#116

Post by lobster_johnson »

apocalyptic0n3 wrote: post #65

Does anyone have any good backup solutions to mitigate this? My agency uses a script we wrote in-house to more-or-less rsync our data to an AWS instance, but it's always seemed a poor way to handle it. I'm using DO's backup service for my personal site which was always supposed to be a temporary solution (the timing of the backups is inconvenient as mentioned by the article). A better solution would be wonderful.

We use Backupninja to drive backups, mostly Duplicity against S3. Some things, like Postgres dumps, are custom commands that just upload individual files to S3. The reason you want to wrap things in Backupninja is to centralize logging, scheduling and monitoring.

Note that we only back up specific things like databases, logs (central syslog server) and data directories. We use Puppet to configure our boxes, and consider everything to be expendable that isn't data.

Re: DigitalOcean lost our server

#117

Post by peterwwillis »

csomar wrote: post #106

Two things to learn from this: 1. Never have a single point of failure. Relying on DO for Server+Backups is putting your eggs on one basket. 2. Your server state should be programmable. This is not quite easy for complex configurations. But today, DO has an API, we have Docker, and quite modern deployment tools. Here is setup: 1. Github for the server state. Basically, a repository to configure and deploy my infrastr…

There's always a single point of failure. Most commonly it's a human.

Re: DigitalOcean lost our server

#118

Post by vardump »

mappu wrote: post #113
vardump wrote: post #103

Earlier quoted context omitted.

That's pretty damn interesting for more traditional hosting type usage. Seems like they sell 50 GB of real SAN backed storage for just 3.6€ per month. How can they afford to be so cheap?

That's not even particularly cheap per-GB - BuyVM ( http://buyvm.net/storage-vps/ ) will sell you 250GB for 7USD/mo. Disclaimer: happy customer

> That's not even particularly cheap per-GB - BuyVM ( http://buyvm.net/storage-vps/ ) will sell you 250GB for 7USD/mo.

Except that is not SAN backed storage.

From the website:

> We only use RAID-60 drive arrays with a minimum of 16 drives

That's somewhat scary. RAID-6 might be almost ok, but striped? No thank you. I bet they don't also have block level checksums.

Re: DigitalOcean lost our server

#119

Post by Dylan16807 »

cortesoft wrote: post #39
gdgtfiend wrote: post #4

This seems like a non-issue to me. If you're using an IaaS provider you should be treating the network as volatile from the get-go. This is the reason AWS has things like auto-scaling groups. You should be designing for failure in "the cloud"

It is no different if you are running your own hardware - your server hardware can fail no matter who owns the machine.

If your server does something weird and corrupts the filesystem, you still have access to it. That can be a huge difference when it comes to restoring the most recent data.
Post reply on HN