Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why the “Digital Ocean killed my company” incident scares the hell out of me

blog.checklyhq.com

Post reply on HN
Print view View original

51–60 of 185 posts

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#51

Post by fhood »

Arguably Raisup should have been better prepared for this eventuality. But, I suspect that many of the people here talking about how their system is infallible made it that way because they, had time and/or money to spare working on it, have a very portable system, or have had something like this happen before.

I feel like HN comments are often sanctimonious to the point of ignorance. A solo dev, scrambling to get his project to the point where there is even the tiniest chance it will succeed, is likely to have a complex network of hardcoded filepaths, hostnames, and other magic numbers, strings, and config files that would make it very difficult to make portable without significant time and effort.

Or maybe the guy was an idiot. But possibly entertain the thought that maybe his situation wasn't exactly comparable to your eh?

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#52

Post by treis »

olau wrote: post #44
Someone1234 wrote: post #31

Earlier quoted context omitted.

Because hypothetically your instance should be hardware agnostic. If the physical hardware dies, it should automatically migrate to another physical server at their data center without your intervention. It will only look like an unexpected restart from your perspective. That's something worth paying for. It would be more comparable to two servers at Hetzner with rapid failover. But even that is more involved since y…

That's wishful thinking. If there's state in that virtual machine, it's probably either stored on the physical host or in a SAN. If it's in the physical host, it has to be fished out of that machine or restored from a backup. If it's a SAN, you can lose your virtual machine if the SAN goes down. I've seen both happen. Actually, a single machine with RAID is surprisingly stable. A provider like Hetzner can switch out…

>If there's state in that virtual machine, it's probably either stored on the physical host or in a SAN

I think the point is to write your application in a way that there's not state in the VM. My VMs are disposable and in fact the way I do deployments is to spin up a new VM in DO and then assign it the floating IP for production. If things go haywire I can easily swap back the IP address to the known good VM.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#53

Post by thruhiker »

mabbo wrote: post #25

It feels like the need for multi-cloud designs and infrastructure are becoming more sensible. If costs are best with one provider, by all means put 95% of your traffic there... But keeping 5% running smoothly on a second cloud, with a simple means to rebalance between them would be a huge win. It would mean the difference between an annoyance (argh, one of my cloud providers doesn't work) and a disaster (my sole clou…

For a business, especially a startup, the opportunity cost often makes multi-cloud infeasible. You'd rather spend that engineering time on delivering features or addressing tech debt. I'm of the opinion that multi-cloud doesn't make much sense if your monthly cloud spend is less than at least $500k.

Pick one of the strong public cloud providers such as AWS, Azure, or GCP and then be judicious about which PaaS offerings you use to avoid unnecessary vendor lock in unless it adds enough value to justify the lock in.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#54

Post by 4ntonius8lock »

terryf wrote: post #32

I am old enough to have lived in soviet times for a short while. One thing I remember from back then - in order to get things done, you had to know someone. Perhaps your aunt's friend worked in the politbureau or your mom's classmate was a friend of the director. Through connections like that you could get what you needed. It was such a relief when times changed and regardless of who you were, you could start exchang…

Personally, I blame the decoupling of the dollar to the gold standard and the distribution of newly minted dollars from the federal reserve into the well connected via banks and corporations that are controlled by a small social segment that all attend the same schools.

Once money became a thing that doesn't cost anything but the changing of zeros, then growth becomes a question not of how to produce something to get money, and but how to get the both the connection and the pedigree needed to receive cheap dollars.

This has created money silos, where the US aristocracy will take hundreds of billions in loses to capture a market and then extract value in monopolistic ways.

You can see this with google, facebook, amazon, etc.

It wasn't always the best companies that won. It was the best companies that had access to the vast capitals pools created out of thin air and who could promise to operate at the monopolistic scales the monied classes were aiming for from the beginning. That is why Ivy leaguers (whether drop outs or not) were chosen as the princelings. They are people who have a lot committed into the system and wouldn't dare betray it: they can be counted on to take things to their logical extreme.

I think also pertinent is the locking out of the middle and lower clases from growth fases of company creation (incentive angel investors and delaying IPOs + legally enforced discrimination against investors based on social class) - Oh and the pooling of legally stolen funds (pensions) into 'safe' stocks. Not to mention the legalization of bribery which has further accelerated our current state of legislative capture.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#55

Post by [deleted] »

Operyl wrote: post #47

When I joined Digital Ocean, back when it was 2-3 months old, I hit a KVM bug in their stack with my network traffic that caused them to rate limit my droplet to 1Kbit/s because it kept taking the host machine down I guess(?), and then them ultimately terminating it without a ticket asking me to even investigate. That was enough to pretty much never consider using Digital Ocean for anything “production” ever again. T…

[deleted]

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#56

Post by pfranz »

duxup wrote: post #36
terryf wrote: post #32

I am old enough to have lived in soviet times for a short while. One thing I remember from back then - in order to get things done, you had to know someone. Perhaps your aunt's friend worked in the politbureau or your mom's classmate was a friend of the director. Through connections like that you could get what you needed. It was such a relief when times changed and regardless of who you were, you could start exchang…

I'll toot my own theory and it is that nobody wants to pay for enough capable support staff, pay to keep the support staff at the ready often enough, pay support staff who want to stay in that role. They want to automate that all away as much as possible. The future is everyone who isn't somebody chatting about how they run their application on X... because they're mysteriously banned form Y and Z and the next guy ta…

I've heard people observe how many support lines have transformed into these impeccably polite, but completely unempowered people--which is more frustrating than being on hold or getting stuck in an automated menu loop because their protocol deflects any anger away from that person (which isn't fair in any circumstance) and leaves you helpless.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#58

Post by therein »

vorticalbox wrote: post #41
jwr wrote: post #22

Earlier quoted context omitted.

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…

If you use a lambda on aws you get 1 million requests for free each month and after that factions of a penny per invoke.

That's wonderful. Now you can use those requests to read from or write to SQS, do DynamoDB stuff and play in the AWS walled garden where things seem cheap at first but things add up.

You can then keep telling yourself you have a stable, managed infrastructure and you're cutting on operational expenses and able to move fast while the engineers in your organization are working harder to make things work with generic services with shortcomings, making your product actually work in less than optimal ways.

Managed services feel like they are helping with operational costs but they have a cost when you're building your product.

I worked on projects that would have ended up being simpler and cheaper to operate on physical dedicated servers, but instead they are running on "ASG's that autoscale and have zero downtime" with "ALBs that send traffic to any and all hosts when all origins report unhealthy".

Things in the managed world is far less than ideal because one size doesn't fit all.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#59

Post by ehnto »

duxup wrote: post #36
terryf wrote: post #32

I am old enough to have lived in soviet times for a short while. One thing I remember from back then - in order to get things done, you had to know someone. Perhaps your aunt's friend worked in the politbureau or your mom's classmate was a friend of the director. Through connections like that you could get what you needed. It was such a relief when times changed and regardless of who you were, you could start exchang…

I'll toot my own theory and it is that nobody wants to pay for enough capable support staff, pay to keep the support staff at the ready often enough, pay support staff who want to stay in that role. They want to automate that all away as much as possible. The future is everyone who isn't somebody chatting about how they run their application on X... because they're mysteriously banned form Y and Z and the next guy ta…

I would support your theory. Support staff don't scale as well or as fast as the technology they're supporting does.

I have often wondered about that for Google. Sure, they have bucket loads of cash, but could they even feasibly stand up a large enough support staff to handle all their platforms? They have so many services, across dozens of languages, and serve hundreds of millions of customers in different timezones.

Also, why would they want to? They're saving untold amounts of money by pushing the problem onto the consumer, and if it works 99% of the time then it's probably good enough.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#60

Post by behringer »

freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

Might I recommend prgmr.com instead? I use both DO and PRGMR (for resiliency).
Post reply on HN