Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

37signals Leaves the Cloud

world.hey.com

Post reply on HN
Print view View original

51–60 of 93 posts

Re: 37signals Leaves the Cloud

#51

Post by flakeoil »

adrianmsmith wrote: post #49

> The cloud excels at two ends of the spectrum ... The first end is when your application is so simple and low traffic that you really do save on complexity by starting with fully managed services .. The second is when your load is highly irregular I think there's a third, which is when the company is so rich it just doesn't matter. Then saving even a little bit of employee time (which are more likely to be the bottl…

Sounds like pure stupidity and incompetence. Why not buy a bunch of Ferrari's and beat them up with sledge hammers while they are at it?

Re: 37signals Leaves the Cloud

#52

Post by flakeoil »

ksec wrote: post #50
wlll wrote: post #43

Earlier quoted context omitted.

*whoops, this got long* Basecamp started on one single Rackspace server, before I was there. I started at 20 people, left at ~55. When I left there were I don't remember how many racks exactly, but more than 10, less than 30 in the primary location. 42U of servers in each. There was a mix of a whole load of (Dell, never pay list price!) blade servers, DB appliances (~12 in total across ~6 apps ISTR), Isilon storage[0…

The reason was that I remember DHH stating they were doing 2K RPS with 30 App Server in 2015 ~ 2016. And they were on one Primary DB. ( At least that was what I jot down in my notes ). I was assuming they could fit multiple "App Server" Node inside a single 1U Blade. But even if it was 1U per App Server, that would be 30U + likely a Powerful 4U DB Monster. Along with probably some Cache instances. Even if the number…

Do you guys define racks the same way?

Re: 37signals Leaves the Cloud

#53

Post by adrianmsmith »

flakeoil wrote: post #51
adrianmsmith wrote: post #49

> The cloud excels at two ends of the spectrum ... The first end is when your application is so simple and low traffic that you really do save on complexity by starting with fully managed services .. The second is when your load is highly irregular I think there's a third, which is when the company is so rich it just doesn't matter. Then saving even a little bit of employee time (which are more likely to be the bottl…

Sounds like pure stupidity and incompetence. Why not buy a bunch of Ferrari's and beat them up with sledge hammers while they are at it?

I think if you're working at a company that makes a huge amount of money, it just isn't really rational to care that much about spend. I think of it like this: If a person had $100m in the bank, I reckon they're not going to care whether their beer costs $7.00 or $7.10. There are just more important things to worry about.

Re: 37signals Leaves the Cloud

#54

Post by BackBlast »

senttoschool wrote: post #48
Nextgrid wrote: post #20

Earlier quoted context omitted.

They are much less frequent, and at the very least you have full access to the hardware to debug if needed. Can you get me access to the AWS control plane's logs so I can understand why my RDS instance is stuck in "modifying" for 12+ hours?

AWS has over a decade of experience and a ton of full-time dev ops to make complicated cloud usage easy. You're not going to beat the reliability by going inhouse.

If something goes wrong at AWS, and it does. You have no power over recovery. The more of the stack you own, the more such ability you do have.

If you run a competent organization, this can be a boon. If you have serious deficiencies, it may well be the opposite.

Re: 37signals Leaves the Cloud

#55

Post by wlll »

ksec wrote: post #50
wlll wrote: post #43

Earlier quoted context omitted.

*whoops, this got long* Basecamp started on one single Rackspace server, before I was there. I started at 20 people, left at ~55. When I left there were I don't remember how many racks exactly, but more than 10, less than 30 in the primary location. 42U of servers in each. There was a mix of a whole load of (Dell, never pay list price!) blade servers, DB appliances (~12 in total across ~6 apps ISTR), Isilon storage[0…

The reason was that I remember DHH stating they were doing 2K RPS with 30 App Server in 2015 ~ 2016. And they were on one Primary DB. ( At least that was what I jot down in my notes ). I was assuming they could fit multiple "App Server" Node inside a single 1U Blade. But even if it was 1U per App Server, that would be 30U + likely a Powerful 4U DB Monster. Along with probably some Cache instances. Even if the number…

The place I work (CTO for now, looking for opportunities pretty soon) can sustain nearly 400 RPS through a Rails app on a single Performance-L Heroku dyno without a sweat (though we run two min), but it's the tip of the iceberg. The infra to support those web servers is way more than that. Aurora Postgres x 3 for now, a large ES cluster, 2x redis clusters, memcached etc.

Bear in mind that we had I think 6 apps. Basecamp 1, 2 and 3 (all separate infra), Highrise, Campfire, some other internal stuff. Our ES cluster was pretty damn big, redis and memcached too. Juniper, F5, network switches (rack infra) etc. Storage was pretty big, quite a few 4u servers with spinning rust. The blade servers were I think 6 blades in 2u.

Everything was redundant, everything had 2x or more. I honestly can't remember how many racks we had. "More than 10" is my hand-wavey guess.

I don't think DHH was being disingenuous with his "2k on 30 servers" message, it was likely more about the scalability of Rails rather than the infra required to run the app.

Re: 37signals Leaves the Cloud

#56

Post by wlll »

flakeoil wrote: post #52
ksec wrote: post #50

Earlier quoted context omitted.

The reason was that I remember DHH stating they were doing 2K RPS with 30 App Server in 2015 ~ 2016. And they were on one Primary DB. ( At least that was what I jot down in my notes ). I was assuming they could fit multiple "App Server" Node inside a single 1U Blade. But even if it was 1U per App Server, that would be 30U + likely a Powerful 4U DB Monster. Along with probably some Cache instances. Even if the number…

Do you guys define racks the same way?

I think so, 42U of full-depth server capacity.

Re: 37signals Leaves the Cloud

#57

Post by codegeek »

senttoschool wrote: post #8

So their AWS bill is $500k/year. Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS? For an email service, you'd think that you want the most reliable host. Weird.

"Hire one person to buy, configure, manage hardware is what? $250k/year? "

Outside of expensive Silicon Valley and FAANGs, where do you get this $250k/Year ? I think most of us on HN are normalizing this whole "devs make 250k/year" thing when in reality, it is a tiny percentage of people. Tiny when you look at the entire ecosystem.

Re: 37signals Leaves the Cloud

#58

Post by pfhayes »

adrianmsmith wrote: post #49

> The cloud excels at two ends of the spectrum ... The first end is when your application is so simple and low traffic that you really do save on complexity by starting with fully managed services .. The second is when your load is highly irregular I think there's a third, which is when the company is so rich it just doesn't matter. Then saving even a little bit of employee time (which are more likely to be the bottl…

This isn’t only true for “rich” companies.

Companies (particularly young ones) succeed or fail if they are innovating on their products and delivering value to customers. Reducing cloud spend is not doing that.

It may very well improve the bottom line, but when the company is (correctly) more focused on “how do we increase our customers by X% this year”, pursuing those savings is a distraction.

Re: 37signals Leaves the Cloud

#59

Post by Nextgrid »

senttoschool wrote: post #48
Nextgrid wrote: post #20

Earlier quoted context omitted.

They are much less frequent, and at the very least you have full access to the hardware to debug if needed. Can you get me access to the AWS control plane's logs so I can understand why my RDS instance is stuck in "modifying" for 12+ hours?

AWS has over a decade of experience and a ton of full-time dev ops to make complicated cloud usage easy. You're not going to beat the reliability by going inhouse.

I'm going to beat its reliability (and successfully do so) because the complexity of my stack is less than 1% of Amazon's. I have way less moving parts (and complex interactions between them) to worry about than Amazon. My only liability is hardware failure, and it turns out that stuff has gotten pretty reliable nowadays.

Am I able to beat Amazon on their wide range of services? Absolutely not - if you're looking for FAANG-level scale, you're talking to the wrong person. But for the handful of services I do need (HTTP LB, app servers, relational database and cache), I beat them because I only need to deliver very specific services and can schedule the risky operations (maintenance, hardware upgrades, etc) around my business' needs, something AWS can't do because as a global provider any time they pick for maintenance would inevitably clash with someone's business-critical hours. Turns out most companies also only require a small chunk of what AWS offers, so the rest of the AWS control plane's complexity and its global scale is actually a major liability.

> AWS has over a decade of experience and a ton of full-time dev ops to make complicated cloud usage easy.

And yet that one time a client's RDS instance ran out of disk space, it got stuck in "modifying" for 12+ hours (I assume an AWS operator eventually manually logged in and fixed it) with no way for us to recover, forcing us to restore a backup and reconstruct missing data from logs to bring our service back online in a timely manner.

Would it fail the same way on bare-metal? Yes, but at least I can SSH in and fix it immediately. Not to mention, the only reason it ran out of space is because AWS' pricing forced us to be very conservative with storage, where as the bare-metal machines I rent - for far cheaper than AWS' offerings - come with terabytes of storage as standard so this incident would've never happened to begin with.

Re: 37signals Leaves the Cloud

#60

Post by Nextgrid »

dijit wrote: post #37
Kon-Peki wrote: post #24

Earlier quoted context omitted.

> Hire one person Go back and re-read :) They [believe that they] don't need to hire anyone. They already have a staff of ops folks that run their cloud operations. And those folks will run their on-prem operations.

Also, Hardware isn't as hard as people make out, you install it once and it runs; if it fails there's a special out-of-band manager built in so you can remotely debug it, datacenters have remote-hands personel. If you're constantly installing stuff then you're scaling up a lot , and that's the level where it would make sense to have someone hired dedicated for it. You can also rent hardware. the thing is: cloud ops a…

> Also, Hardware isn't as hard as people make out, you install it once and it runs; if it fails there's a special out-of-band manager built in so you can remotely debug it, datacenters have remote-hands personel.

If you're "cloud native" you've already architected your stack to be horizontally-scalable and distributed (partly because the cloud is so expensive that running at peak load capability all the time would bankrupt you) so hardware failure should matter even less.

Post reply on HN