Live data from Hacker News

11 years of hosting a SaaS

ghiculescu.substack.com

181–190 of 222 posts

Re: 11 years of hosting a SaaS

#181
post #111

> But the worst Digital Ocean incident we ever had was when they turned all our droplets off all at once. The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. So for probably a month we were getting and ignoring billing alerts, until we really paid attention when everything was offline and not responding to…

I'd say that your service is dead unless proven otherwise. Your bills are unpaid unless proven otherwise. I'd be alerted by absence of regular emails telling me that my card was billed for the hosting service.

> I'd say that your service is dead unless proven otherwise.

I'd say you're dead right -- this is the only safe assumption. And any email client smart enough to check incoming email against a regexp could be extended to raise a not-easily-ignored alert if no pattern match (indicating success in processing a payment) had arrived by some chosen date of the month. Yet I've never come across such a feature in Gmail, Fastmail, nor Thunderbird. Maybe someone sometime has hacked such a thing into Gnus ...

Re: 11 years of hosting a SaaS

#182
post #112

Earlier quoted context omitted.

Does Cloud 66 have any resources on migrating a large (~300 GB) Postgres DB off Heroku, with minimal downtime? This has been our biggest sticking point when thinking about leaving Heroku.

Moving data is always the trickiest part! We get a lot of Heroku customers who need help with their data migration. My honest answer is there will always be some downtime while the switch over is happening, but we know of a few ways to reduce it. Unfortunately Heroku PG databases don't allow outside replication setup, so we can't really baseline your data and then close the replication gap with a shorter downtime. Bu…

We'll keep that in mind, we're currently locked into a (ridiculous) enterprise contract with them. But it expires in November :)

We're considering using this https://bucardo.org/Bucardo/ based on this https://www.porter.run/blog/migrating-postgres-from-heroku-t...

Going to give it a shot when I have some free time.

Re: 11 years of hosting a SaaS

#183

> But the worst Digital Ocean incident we ever had was when they turned all our droplets off all at once. The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. So for probably a month we were getting and ignoring billing alerts, until we really paid attention when everything was offline and not responding to…

Whatever happened to companies actually reaching out with a phone call to an actual human in order to resolve these issues before they become a problem? A provider that simply throws a switch because a credit card expired is not one that I would trust. So, yeah, DO (and every other provider that operates that way) deserves a little bit of shade for actions that clearly demonstrate a lack of care for their customers.

Re: 11 years of hosting a SaaS

#184

Would be interested to know what your hosting bill was like? We recently moved everything in-house and started hosting our own servers. Having physical access to the machine (and the network HW) pretty much eliminated all of our dev ops (1). We expected the HW to fail often and such, but in 2 years the only maintenance we had to do was add more disks to the RAID array and resize the partition a few times. Even though…

> since we owned the hardware it meant that efficient utilization wasn't a requirement for us anymore This is a great point. Hyper focus on optimizing micro-containers is costly and sucks up a lot of time. You can still deploy like this in the cloud though. On AWS make a three-year reservation of a c6a.32xlarge to get 128 vCPU for $16k annually. Hetzner is cheaper, 80 vCPU for about €2,506 annually. [1] https://insta…

This is the way. Unless you are huge, self hosting and even colo is not competitive with long term VM commitments in hyper scalers.

They get big discounts on everything you'd pay retail, and their margins for barebones VM is not that large.

Re: 11 years of hosting a SaaS

#185

Earlier quoted context omitted.

I don’t think it’s unreasonable to expect that multiple channels of communication will be tried before cutting off service, especially not for services provided to a business.

For a business that makes checks notes 50 bucks a month in revenue with DO? If this were the case, prices would be astronomically high because there would be tons of scammers and bad actors abusing this system, leaving their servers running for months, because apparently DO has to send them a letter via snail mail to Antarctica and wait for their response. All of this would then have to be paid for by legitimate cust…

Also, the more low level customer support workers you add to your cloud service, the more insecure it becomes, and typically the worse the support experience becomes (hiring people who actually have a good handle of how cloud software works ain’t cheap).

If you have an account manager or sales folk in the cloud org, IOW have a high spend, you get the twitter experience. If you don’t, and are a self-serve customer spending a pretty low amount like in this case and missed a payment and ignored the account emails reminding you to pay… what exactly is the provider supposed to do? I mean this in the nicest way possible but they aren’t so desperate for your business that they’re going to beat down your door so you can pay before you get shutdown. Important to keep in mind, if you don’t have a long payment history, you don’t look that different from a user that’s just trying to get something for free, and without paying for the communication channel, you also can resemble a company that just went bankrupt or something (which often creates a huge risk for the cloud provider if the cloud creds get sold by some disgruntled ex employee to crypto miners).

Source: I worked in this area at a major public cloud provider

Re: 11 years of hosting a SaaS

#186
post #14

the progression from Heroku to AWS is very common as SaaS businesses grow. However one thing that’s missing from all “alternatives to Heroku” is what I call life after deployment or Day-2 operations. Many solutions show you a sleek demo of how easily you can deploy a Hello World app to their platform but leave you in a straight jacket when it the app is deployed and you need tools to keep it running, like changing an…

I hate what Salesforce has done to Heroku but after a year of running our saas on aws, render, and fly - here I am, back on Heroku. It sucks, but slightly less than the alternatives.

Fly has potential, but they've changed/grown so much that most docs are out of date, everything is buggy, support is not very responsive, and their security posture leaves a lot to be desired. Not a fun place to be production issues pop up.

Re: 11 years of hosting a SaaS

#187
post #14

the progression from Heroku to AWS is very common as SaaS businesses grow. However one thing that’s missing from all “alternatives to Heroku” is what I call life after deployment or Day-2 operations. Many solutions show you a sleek demo of how easily you can deploy a Hello World app to their platform but leave you in a straight jacket when it the app is deployed and you need tools to keep it running, like changing an…

I hate what Salesforce has done to Heroku but after a year of running our saas on aws, render, and fly - here I am, back on Heroku. It sucks, but slightly less than the alternatives. Fly has potential, but they've changed/grown so much that most docs are out of date, everything is buggy, support is not very responsive, and their security posture leaves a lot to be desired. Not a fun place to be production issues pop…

I'm building Argonaut to be a layer on top of AWS/GCP to make them less sucky. The primary use case is startups that have a bunch of services and have outgrown heroku etc. I'd love to chat if we can help - you get the stability of AWS and a push to deploy dev experience for your apps.

Re: 11 years of hosting a SaaS

#188

Earlier quoted context omitted.

They literally send an email every single day for a month. And they are flexible enough that if you do write them they will happily extend that grace period. It’s like saying car is a shitty transportation service because it stops working once the gas runs out. Not paying attention the the reserve light, instrument msg and pings is solely your problem. More so because you are running a business…

The car should arrange for a singing telegram to be delivered to your home, office AND your child’s daycare to notify you before shutting off its engine due to lack of fuel. In my country this is required by law.

Hilarious comment. Thanks for brightening my day.

Re: 11 years of hosting a SaaS

#189

> But the worst Digital Ocean incident we ever had was when they turned all our droplets off all at once. The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. So for probably a month we were getting and ignoring billing alerts, until we really paid attention when everything was offline and not responding to…

Whatever happened to companies actually reaching out with a phone call to an actual human in order to resolve these issues before they become a problem? A provider that simply throws a switch because a credit card expired is not one that I would trust. So, yeah, DO (and every other provider that operates that way) deserves a little bit of shade for actions that clearly demonstrate a lack of care for their customers.

They were paying all of $40 a month, I'd imagine that doesn't buy them very much customer support from any decently sized cloud provider.

Re: 11 years of hosting a SaaS

#190
post #12

> Use managed services for as long as possible Big agree here. Yes, you can save stupid money by handrolling postgres on an extremely beefy Hetzner server, or you can pay someone else and keep building your product: https://onlineornot.com/self-hosting-vs-managed-services-dec... This isn't to say, "don't bother learning how to do it yourself", but more "learn to pick your battles".

Which one do you recommend? Those get quite expensive.

I personally use RDS
Post reply on HN