Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Build vs. Buy: What This Week's Outages Should Teach You

toddhgardner.com

Post reply on HN
Print view View original

31–40 of 46 posts

Re: Build vs. Buy: What This Week's Outages Should Teach You

#31

Post by wrs »

dan353hehe wrote: post #22

> Here’s the thing, if your core business function depends on some capability, you should own it if at all possible. If I'm building something that allows my customers to do X, then yes I will own the software that allows my customers to do X. Makes sense. > They’ll craft artisanal monitoring solutions while their actual business logic—the thing customers pay for—runs on someone else’s computer. So instead I should b…

The OP's point is that if your monitoring solution dies, your customers don't even notice, so you shouldn't build it yourself. But if the service running your actual business logic dies, your customers get cut off, so you should build and maintain that part more directly. (And obviously this is a spectrum — you probably don't need to design your own CPU.)

Re: Build vs. Buy: What This Week's Outages Should Teach You

#32

Post by 1970-01-01 »

toddgardner wrote: post #5
1970-01-01 wrote: post #2

Meh. This opinion highlights the fact that availability is the least understood pillar in security. The Right Way to Think About It is having good security analysis and doing proper Risk Management. This means it is their job to do business impact analysis, 3rd party assessments, and run tabletop exercises on all your critical systems to tell you what is rock solid and what is a house of cards.

How you approach this is very different depending on the size of organization. We're a small shop (3), but we deliver big services to lots of people. We do this by owning everything we can, and using simple vendors for what we can't.

Understanding exactly who does what and how they can be reached to work an emergency is all part of the availability pillar. Size matters not. Your security team needs to vet your team, your critical systems, your code, and your 3rd and 4th party dependencies constantly.

Re: Build vs. Buy: What This Week's Outages Should Teach You

#33

Post by dylan604 »

mannyv wrote: post #13

What this outage teaches you is that when a third party vendor fails and the internet breaks you can point the finger at them with no issues. If your shit breaks and everyone else's shit is still working that's a problem.

any company offering services with SLA that does not have this as a caveat is just crazy to me. "we guarantee our services will be up and running as long as the 3rd party services we run on top of are running."

Re: Build vs. Buy: What This Week's Outages Should Teach You

#34

Post by onion2k »

wrs wrote: post #31
dan353hehe wrote: post #22

> Here’s the thing, if your core business function depends on some capability, you should own it if at all possible. If I'm building something that allows my customers to do X, then yes I will own the software that allows my customers to do X. Makes sense. > They’ll craft artisanal monitoring solutions while their actual business logic—the thing customers pay for—runs on someone else’s computer. So instead I should b…

The OP's point is that if your monitoring solution dies, your customers don't even notice, so you shouldn't build it yourself. But if the service running your actual business logic dies, your customers get cut off, so you should build and maintain that part more directly. (And obviously this is a spectrum — you probably don't need to design your own CPU.)

if the service running your actual business logic dies

In a modern tech business that's everything from the frontend to the database though, including all the bits to keep that running at scale. That's too much for most companies to handle when they're starting and scaling. You'll need to compromise on that value early on, and you'll probably persuade yourself that it's tech debt you'll pay off later. But you won't, because you can't, and that will lead you to dislike the system you built.

It's much simpler and more motivating to accept that in any modern tech business has to rely on third parties, and the fact you pay them money means they probably won't screw it up. It has to be an accepted risk or you'll be paralysed by having too much to do.

Re: Build vs. Buy: What This Week's Outages Should Teach You

#35

Post by toddgardner »

mannyv wrote: post #13

What this outage teaches you is that when a third party vendor fails and the internet breaks you can point the finger at them with no issues. If your shit breaks and everyone else's shit is still working that's a problem.

I tend to sell to a wide variety of customers. They tend not to give a crap if a cloud provider is down, its still our problem to make it right.

Re: Build vs. Buy: What This Week's Outages Should Teach You

#36

Post by gwbas1c »

vivzkestrel wrote: post #12

Instead we need a startup that builds over every cloud provider. Think of a web server for example. AWS has EC2, GCP has its own equivalent and Azure has its own and so on. What if we had a startup that virtualizes a layer on top of these such that we AWS has an outage, you lose 1/3rd of your operating capacity, when Azure has an outage you lose 1/3rd of your operating capacity. In order for you startup s virtual web…

> What if we had a startup that virtualizes a layer on top of these such that we AWS has an outage, you lose 1/3rd of your operating capacity, when Azure has an outage you lose 1/3rd of your operating capacity.

And then when your startup goes down we lose 3/3rds of our operating capacity!

---

There are certain kinds of errors and failures that it's not worth protecting against, because the costs (and consequences) are more than just accepting that things fail from time to time.

It's easy to forget that services used to go down all the time in the 1990s and early 2000s. In this case, we still have super-impressive resiliency with modern cloud hosting.

IMO: The best way to improve the situation is for the cloud hosts to take their lessons learned and improve themselves, and for us (their customers) to vote with our feet if/when a cloud provider has problems.

Re: Build vs. Buy: What This Week's Outages Should Teach You

#37

Post by gwbas1c »

> if your core business function depends on some capability, you should own it

I fear this is easy to misconstrue.

For example, I was at a company that, as I learned how everything worked, realized that we were spending $20k / month for cloud services to basically process about as much real-time data as a CD player processes.

I joked that we should be able to run our entire product on a single server running in the office. (Then I pointed out that this was a joke and that running in the cloud gave us amazing redundancy that we didn't have to implement ourselves.) My point was to show that our architecture was massively bloated and overengineered for what we were doing. (IE, the cost of serialization to send messages was more than the actual processing that was happening. The cost was both money, and the fact that we were spending more time working on messaging than the actual product.)

BUT: There's many times where we could easily say, "this would be so much easier if we had our own server in the office." And, if we misconstrue the above quote, we could convince ourselves to run our own server in the office.

Re: Build vs. Buy: What This Week's Outages Should Teach You

#38

Post by toddgardner »

gwbas1c wrote: post #37

> if your core business function depends on some capability, you should own it I fear this is easy to misconstrue. For example, I was at a company that, as I learned how everything worked, realized that we were spending $20k / month for cloud services to basically process about as much real-time data as a CD player processes. I joked that we should be able to run our entire product on a single server running in the o…

Yea totally. this is a balance.

Very few times should you manage the actual hardware yourself.

But often a cloud is overly complex for what you need. 10 years ago we left MS Azure and started leasing dedicated hardware in OVH. Our costs were cut by 90%, our performance tripled, and our reliability improved. We did have to take on some effort to make our systems portable with ansible and containers, but we greatly simplified our vendor stack.

I am never confused why something goes down, and I have confidence that I can stand up with another vendor without re-writing anything.

If I can't own it, it should be as simple and commoditized as possible. Most clouds are not that.

Re: Build vs. Buy: What This Week's Outages Should Teach You

#40

Post by erikpukinskis »

If I build and my own CDN, it will go down. And I will have to fix it at 2am.

If I use CloudFlare, it will also go down, but probably for less time, and someone else has to be up at 2am fixing it.

> Build what delivers your value.

Like Hershey builds grocery stores?

Like Budweiser builds bars?

This can’t be serious.

We live in a society.

Post reply on HN