37signals Leaves the Cloud • Hacker News Board

Re: 37signals Leaves the Cloud

#61

Post by jjav » Thu, Oct 20, 2022, 11:38 PM UTC

senttoschool wrote: post #48

Nextgrid wrote: post #20

Earlier quoted context omitted.

They are much less frequent, and at the very least you have full access to the hardware to debug if needed. Can you get me access to the AWS control plane's logs so I can understand why my RDS instance is stuck in "modifying" for 12+ hours?

AWS has over a decade of experience and a ton of full-time dev ops to make complicated cloud usage easy. You're not going to beat the reliability by going inhouse.

> You're not going to beat the reliability by going inhouse.

OP was talking about observability and diagnosability. These will be infinitely better/easier with access to the hardware & OS.

But on reliability, if you keep your stack simple and boring, you can pretty easily also beat them on reliability. Complexity is the top enemy of reliability (and security).

Re: 37signals Leaves the Cloud

#62

Post by jjav » Fri, Oct 21, 2022, 12:50 AM UTC

senttoschool wrote: post #8

So their AWS bill is $500k/year. Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS? For an email service, you'd think that you want the most reliable host. Weird.

> So their AWS bill is $500k/year.

> Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS?

A company which has a total AWS bill of $500k/yr is a very small one (the article says 500K is just the RDS/ES bill, so total is surely much higher for them).

Our (tiny startup) AWS bill is ~200K/yr and the amount of compute and storage we do could very easily be handled by one single physical server.

Now, sure, if we went to self-hosted we'd split that into multiple servers for redundancy and growth headroom, but in any case definitely less than a dozen boxes.

You don't hire anyone to install and manage the hardware at that scale. Someone from the ops team goes to the colo for a day or two to set it all up and then you only need to go back when something needs replacement, which at that small scale is going to be very rarely.

That's exactly what we did at a previous startup and the company grew to many, many hundreds of people before there was so much infrastructure that it became worth thinking about hiring a dedicated person. By that time you'd be in the 10s of millions in AWS spend.

Re: 37signals Leaves the Cloud

#63

Post by uvdn7 » Fri, Oct 21, 2022, 1:19 AM UTC

Managing machine life cycles, data intensive and stateful systems is very hard. I was hoping to see more concrete examples/data points instead of just saying we can also run the same software on our own machines.

Re: 37signals Leaves the Cloud

#64

Post by pikseladam » Fri, Oct 21, 2022, 6:49 AM UTC

senttoschool wrote: post #8

So their AWS bill is $500k/year. Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS? For an email service, you'd think that you want the most reliable host. Weird.

You don't need to hire.

Re: 37signals Leaves the Cloud

#65

Post by wazoox » Fri, Oct 21, 2022, 8:08 AM UTC

mattbrewsbytes wrote: post #9

Very valid points. Its hard to describe the complexities and weird error or latency or other conditions that happen in cloud environments. OPs people kinda shrug and ask that you essentially turn it off and back on again and that usually resolves the issue at that time. There are so many abstraction layers when all the infrastructure is software run and I bet a lot of those introduce performance degradation. Another…

Ah yeah, I remember when a friend's company struggled for weeks with some terrible slowdowns on their Azure-based cloud infrastructure. After hours on hours of support calls and emails and weeks of hair-pulling, everything suddenly went better : after careful interviews with Azure support team, they understood that their VMs were running on hosts in two different racks, and that the link between the racks had just been upgraded from gigabit to 10-gig when the ToR switches were replaced.

Re: 37signals Leaves the Cloud

#66

Post by wazoox » Fri, Oct 21, 2022, 8:12 AM UTC

jjav wrote: post #62

senttoschool wrote: post #8

So their AWS bill is $500k/year. Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS? For an email service, you'd think that you want the most reliable host. Weird.

> So their AWS bill is $500k/year. > Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS? A company which has a total AWS bill of $500k/yr is a very small one (the article says 500K is just the RDS/ES bill, so total is surely much higher for them). Our (tiny startup) AWS bill is ~200K/yr and the amount of compute and storage we do could very easily b…

Also nowadays hardware as well as software are incredibly reliable. You can basically set up a Debian web + postgres server with automatic updates enabled and not need to even log into it for months and months.

Re: 37signals Leaves the Cloud

#67

Post by pelasaco » Fri, Oct 21, 2022, 8:32 AM UTC

senttoschool wrote: post #8

So their AWS bill is $500k/year. Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS? For an email service, you'd think that you want the most reliable host. Weird.

Person + data center, right? Either he will build his own small server room or have to do co-location somewhere..

Re: 37signals Leaves the Cloud

#68

Post by Abe9000 » Fri, Oct 21, 2022, 10:21 AM UTC

pelasaco wrote: post #67

senttoschool wrote: post #8

So their AWS bill is $500k/year. Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS? For an email service, you'd think that you want the most reliable host. Weird.

Person + data center, right? Either he will build his own small server room or have to do co-location somewhere..

WIth 500K yearly you can do a lot!! Including renting 3 office spaces in multiple locations in the US and using them as data centers. Offices cost about $1000 - $3000 yearly, he doesn't need a premium space in a big building but for $500K yearly you can do a lot! In 2 years you would have spent 1M. That is more than enough to have a dedicated devops that travels to all the locations and setup the super beefy servers you can buy with that budget.

Re: 37signals Leaves the Cloud

#69

Post by Abe9000 » Fri, Oct 21, 2022, 10:23 AM UTC

vikeri wrote: post #2

Curious to know how to which degree they are self hosting, seems they'll buy their own servers but I doubt they'll go all the way to buying land and building their own data centers?

Who told you they need to build a massive data center? They are NOT RENTING SERVERS but buying space for the ones they use. With the budget they can afford to buy super beefy rack servers that occupy no space. With a single rack they can have a ton of late tech servers.

Re: 37signals Leaves the Cloud

#70

Post by Abe9000 » Fri, Oct 21, 2022, 10:25 AM UTC

DEATH TO THE CLOUD. For far too long they have profited out of the ignorance of those who do not want to learn how to virtualize, or host their own site. Have you seen the specs of the server you rent? A 2 GHZ core? Really? Time to spend real money in a real hardware. Fiber optic has dropped latency to really low levels. You don't need a super close geolocation server anymore. With the money they spend on the cloud, they can rent small offices all over the world and have a small rack server with beefy servers in a secure server room they can build.