Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DigitalOcean block storage is down

status.digitalocean.com

Post reply on HN
Print view View original

51–60 of 88 posts

Re: DigitalOcean block storage is down

#51

Post by pastrami_panda »

zeta0134 wrote: post #47
pastrami_panda wrote: post #44

This is OT, but I have a droplet on DO and I'm amazed at the amount of malicious traffic it gets. Is it normal for a very private vps to receive thousands of ssh attempts per hour? I have fail2ban installed and the jail is so busy it's quite astounding. Anyone with more web hosting experience that can weigh in?

I work for a web hosting company in Texas, and this is ridiculously common. Any public IP with any public service at all will be poked, prodded, and generally made uncomfortable by every bot and crawler you can think of, trying common password combinations and scanning for common vulnerabilities in popular software. This catches so many of our customers by surprise, who tend to mistakenly believe they're being target…

Cheers for weighing in. A whitelist is a good solution, since the sheer amount of attempts is making me uncomfortable. It seems to be accelerating over time as well which is even more disturbing.

Re: DigitalOcean block storage is down

#52

Post by stephenr »

Bombthecat wrote: post #50
stephenr wrote: post #35

This is your weekly reminder that anything you want to be reasonably “HA” should span multiple vendors in multiple DCs.

Yeah, the myth that "just use aws" to have 99, 9999999 percent uptime is coming to an end...

Oh I'm sure the myth will persist for many years.

Re: DigitalOcean block storage is down

#53

Post by dc352 »

nonbirithm wrote: post #48
kyledrake wrote: post #2

What unholy thing did they do that broke it across 12 different datacenters, good lord.

It could be DNS. Azure has had an all-region failure due to a single DNS provider outage. It was possible that same DNS provider's outage was also causing problems for GCE and AWS at the same time. https://news.ycombinator.com/item?id=19812919

That wouldn't be at the top of my list. We have "Volumes" for databases and they were inaccessible for like 6 hours. I don't think any DNS is involved in mounting these. But hey, there's always a lot of crap hidden behind the scenes :)

Re: DigitalOcean block storage is down

#54

Post by mercer »

pastrami_panda wrote: post #44

This is OT, but I have a droplet on DO and I'm amazed at the amount of malicious traffic it gets. Is it normal for a very private vps to receive thousands of ssh attempts per hour? I have fail2ban installed and the jail is so busy it's quite astounding. Anyone with more web hosting experience that can weigh in?

I had the same experience on DO as well as a few other providers.

Changing the ssh port to something in the 50_000 range drastically reduced the number of attempts and left my logs much cleaner :).

Re: DigitalOcean block storage is down

#55

Post by dc352 »

stephenr wrote: post #35

This is your weekly reminder that anything you want to be reasonably “HA” should span multiple vendors in multiple DCs.

that would be pretty cool but to have that, you need a high-network-latency solution, i.e., pretty much cold back-up. For some time I thought it's pretty last century option but having been experimenting for some time now, it's the option with lowest impact on system performance. More importantly, it's reasonably resilient.

Re: DigitalOcean block storage is down

#56

Post by dc352 »

sondh wrote: post #19
golanggeek wrote: post #3

This is really down for more than 2 hours!!!

Last night I was testing DO managed Kubernetes cluster with persistent volume claim and the volume took 15 minutes to reattach after the pod is rescheduled to another host. I thought it was just some weird hiccup and went to bed. The incident report indicated the problem started 4 hours ago (around 9pm GMT) but I was having problem around 4pm. It's definitely not a 2-hour incident.

our disks in London went down at about 8:45pm UTC (10 mins 100% disk utilization alert triggered at 5 to) and DO recovery message was sent out at about 2am UTC. We switched our service (keychest.net) on at 3:15am

Re: DigitalOcean block storage is down

#57

Post by stephenr »

dc352 wrote: post #55
stephenr wrote: post #35

This is your weekly reminder that anything you want to be reasonably “HA” should span multiple vendors in multiple DCs.

that would be pretty cool but to have that, you need a high-network-latency solution, i.e., pretty much cold back-up. For some time I thought it's pretty last century option but having been experimenting for some time now, it's the option with lowest impact on system performance. More importantly, it's reasonably resilient.

I've read your comment now about 4 times and all I have come up with is "huh?"

Literally thousands if not millions of organisations operate multi-DC infrastructure across the planet.

Is it harder than setting up a single box in one DC? Yes. Is it harder than setting up a mini-cluster of boxes in one DC? Yes. Is it rocket science? No.

Re: DigitalOcean block storage is down

#58

Post by ngrilly »

seaghost wrote: post #36

Their block storage is such a failure. I’m back and forth with support to automatically delete files with lifecycles for over 2 months now and it’s still not resolved.

Since you're trying to "delete file with lifecycles", I'm quite sure your problem is with their object storage (called Spaces), and not their block storage.

Re: DigitalOcean block storage is down

#59

Post by ngrilly »

louwrentius wrote: post #41

Isn't Digital Ocean running Ceph for their block storage? I would wonder - as others suggested - that they may have stretched the cluster across datacenters ?! Would be interested in the post-mortem.

Yes, DO uses Ceph: https://blog.digitalocean.com/why-we-chose-ceph-to-build-blo...

Re: DigitalOcean block storage is down

#60

Post by markonen »

dc352 wrote: post #53
nonbirithm wrote: post #48

Earlier quoted context omitted.

It could be DNS. Azure has had an all-region failure due to a single DNS provider outage. It was possible that same DNS provider's outage was also causing problems for GCE and AWS at the same time. https://news.ycombinator.com/item?id=19812919

That wouldn't be at the top of my list. We have "Volumes" for databases and they were inaccessible for like 6 hours. I don't think any DNS is involved in mounting these. But hey, there's always a lot of crap hidden behind the scenes :)

I would be absolutely amazed if DNS was not involved in mounting a block storage volume.
Post reply on HN