Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

11 years of hosting a SaaS

ghiculescu.substack.com

Post reply on HN
Print view View original

61–70 of 222 posts

Re: 11 years of hosting a SaaS

#61

Post by withinboredom »

AndroTux wrote: post #59

> But the worst Digital Ocean incident we ever had was when they turned all our droplets off all at once. The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. So for probably a month we were getting and ignoring billing alerts, until we really paid attention when everything was offline and not responding to…

Not exactly. Some countries have laws in place that prevent a provider from removing service due to non-payment to a business. Instead they must follow a process, including physical paper notices, before removing service.

DO should have done something like that as well, just cutting off service because you sent some emails is kinda shitty.

Re: 11 years of hosting a SaaS

#62

Post by chillfox »

AndroTux wrote: post #59

> But the worst Digital Ocean incident we ever had was when they turned all our droplets off all at once. The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. So for probably a month we were getting and ignoring billing alerts, until we really paid attention when everything was offline and not responding to…

I don’t think it’s unreasonable to expect that multiple channels of communication will be tried before cutting off service, especially not for services provided to a business.

Re: 11 years of hosting a SaaS

#63

Post by marginalia_nu »

maccard wrote: post #58
marginalia_nu wrote: post #48

Earlier quoted context omitted.

A lot of "modern" stacks is just complexity for the sake of complexity. Google is doing it so clearly our 5 man startup will face the same scaling problems, or something like that. Many of the problems these tools solve are problems that wouldn't exist building things the old fashioned way. If you stick relatively close to the metal, operating this stuff is pretty easy. However it's notable that a very valid reason t…

The complex stacks are insane to operate yourself but very simple to operate if you use a managed offering instead, and they do provide genuine value. I can set up a new golang app on ECS with a load balancer and database, with a CI/CD pipeline, with 0 downtime updates in about 30 minutes. Most of that time is waiting for AWS to give me a load balancer. Our work applications have been running with this setup for over…

To be fair, I can set up a new service on bare metal in minutes too, mostly because I don't need to set up everything from scratch.

I don't really need to set up a database or load balancer or anything like that because it already exists on the server. Just create a new database schema, new systemd service, new nginx rule.

Re: 11 years of hosting a SaaS

#64

Post by MattGaiser »

AndroTux wrote: post #59

> But the worst Digital Ocean incident we ever had was when they turned all our droplets off all at once. The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. So for probably a month we were getting and ignoring billing alerts, until we really paid attention when everything was offline and not responding to…

> The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored.

The using of an unmonitored email is the one that really gets to me.

Re: 11 years of hosting a SaaS

#65

Post by onion2k »

withinboredom wrote: post #61
AndroTux wrote: post #59

> But the worst Digital Ocean incident we ever had was when they turned all our droplets off all at once. The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. So for probably a month we were getting and ignoring billing alerts, until we really paid attention when everything was offline and not responding to…

Not exactly. Some countries have laws in place that prevent a provider from removing service due to non-payment to a business. Instead they must follow a process, including physical paper notices, before removing service. DO should have done something like that as well, just cutting off service because you sent some emails is kinda shitty.

DO should have done something like that as well, just cutting off service because you sent some emails is kinda shitty.

DO should abide by the law of the country they operate in. If that means they can shut off services for non-payment then that's what they're entitled to do. By the sounds of it they gave plenty of warning and 30 days service they weren't getting paid for.

Email is the expected form of business communication in 2023. I don't think any of the hosts I use even have my physical address. Maybe they could SMS me. I doubt they have my phone number either though.

I would expect a billing warning notice when I SSH into the service though. Maybe that was in place for the 30 days before they shut the customer out. I certainly hope so.

Re: 11 years of hosting a SaaS

#66

Post by withinboredom »

aetherspawn wrote: post #10

Would be interested to know what your hosting bill was like? We recently moved everything in-house and started hosting our own servers. Having physical access to the machine (and the network HW) pretty much eliminated all of our dev ops (1). We expected the HW to fail often and such, but in 2 years the only maintenance we had to do was add more disks to the RAID array and resize the partition a few times. Even though…

IME, if a drive is going to fail, it's going to fail in the first hundred hours or so of use. In my entire life (of using hundreds of disks), I've only had three failures. Two of those were on the same server, at the same time, in the same RAID array and took out the whole array -- the data was gone forever. Thank the gods for backups.

So, just make sure you take backups pretty regularly because you never know.

Re: 11 years of hosting a SaaS

#67

Post by ghiculescu »

MattGaiser wrote: post #64
AndroTux wrote: post #59

> But the worst Digital Ocean incident we ever had was when they turned all our droplets off all at once. The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. So for probably a month we were getting and ignoring billing alerts, until we really paid attention when everything was offline and not responding to…

> The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. The using of an unmonitored email is the one that really gets to me.

Yeah. I'm not proud of it.

Re: 11 years of hosting a SaaS

#68

Post by auggierose »

withinboredom wrote: post #61
AndroTux wrote: post #59

> But the worst Digital Ocean incident we ever had was when they turned all our droplets off all at once. The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. So for probably a month we were getting and ignoring billing alerts, until we really paid attention when everything was offline and not responding to…

Not exactly. Some countries have laws in place that prevent a provider from removing service due to non-payment to a business. Instead they must follow a process, including physical paper notices, before removing service. DO should have done something like that as well, just cutting off service because you sent some emails is kinda shitty.

What countries are those?

Re: 11 years of hosting a SaaS

#69

Post by ghiculescu »

onion2k wrote: post #65
withinboredom wrote: post #61

Earlier quoted context omitted.

Not exactly. Some countries have laws in place that prevent a provider from removing service due to non-payment to a business. Instead they must follow a process, including physical paper notices, before removing service. DO should have done something like that as well, just cutting off service because you sent some emails is kinda shitty.

DO should have done something like that as well, just cutting off service because you sent some emails is kinda shitty. DO should abide by the law of the country they operate in. If that means they can shut off services for non-payment then that's what they're entitled to do. By the sounds of it they gave plenty of warning and 30 days service they weren't getting paid for. Email is the expected form of business commu…

I SSH'd plenty. There was no billing notice.

I think that + allowing customers to add more than one credit card to an account are reasonable suggestions. Maybe they do that now - this was 7 years ago.

I'm not trying to shirk responsibility though. We didn't, for example, pass the buck by telling our customers our site was offline because of DO.

Post reply on HN