The Servers Are Burning • Hacker News Board

Re: The Servers Are Burning

#21

Post by adrianN » Thu, Sep 06, 2018, 7:00 PM UTC

noelwelsh wrote: post #10

How could such a tiny change have such an outsized impact on the site? “That same story happened so many different times,” my old boss David told me. “Someone launched a small, relatively innocuous change that did one of the millions of unexpected things it could have done, which then happened to break some part of the site, and then bring it all down—sometimes bring it down to the point where we couldn’t recover it…

A slight change in performance characteristics can also lead to cascading failures. Pure functions don't help with that. That aside, even functional programs need to have side effects (other than heating up the CPU). Referential transparency doesn't help when the state of your monad is a little bit weird and that causes some corner of your code to kill the DB.

Re: The Servers Are Burning

#22

Post by theseatoms » Thu, Sep 06, 2018, 7:00 PM UTC

stcredzero wrote: post #14

blacksmith_tb wrote: post #12

Earlier quoted context omitted.

I assume the implication is "if OKC can fly by the seat of their pants and still make millions, anyone could!" Which is true, but still not advisable...

Given that technical debt is quite hard for non-technical people to quantify, why wouldn't we expect that most startups would tend to accrue technical debt?

Can anyone quantify technical debt? Best case is something resembling a medical diagnosis.

Re: The Servers Are Burning

#23

Post by drblast » Thu, Sep 06, 2018, 7:02 PM UTC

The reactions to the outage on Twitter are hilarious and make me glad this happened:

“@okcupid how am I supposed to get my daily dose of crushing rejection and emotional humiliation if your site is down????”

“Okcupid stops working over lunch hour, NYC wonders if we're committed to the people in our phones from now on, panic in the streets”

“@okcupid How can I continue to be ignored by the females of the world if they don't know where I am to ignore me?! #panic #freakout”

Re: The Servers Are Burning

#24

Post by adrianN » Thu, Sep 06, 2018, 7:03 PM UTC

stcredzero wrote: post #14

blacksmith_tb wrote: post #12

Earlier quoted context omitted.

I assume the implication is "if OKC can fly by the seat of their pants and still make millions, anyone could!" Which is true, but still not advisable...

Given that technical debt is quite hard for non-technical people to quantify, why wouldn't we expect that most startups would tend to accrue technical debt?

Startups accrue so much financial debt, adding some technical debt on top doesn't really make a difference.

Re: The Servers Are Burning

#25

Post by swalsh » Thu, Sep 06, 2018, 7:03 PM UTC

JackFr wrote: post #2

Wow. 1. We don't test. 2. We don't code review (or rather if we do, we do it so poorly swallowed exceptions don't raise red flags.) That's an outrageously unprofessional software process.

It's a free dating service. They have a responsibility to respect their users' data privacy, but the rest kind of doesn't matter. Eventual stability is more profitable than eventual release if you're not a Bank or making self driving cars.

Re: The Servers Are Burning

#26

Post by ahallock » Thu, Sep 06, 2018, 7:04 PM UTC

> ...in order to write effective tests, a programmer had to know all of the ways that a piece of software could fail in order to write tests for those cases

That's impossible. You reason about the problem as best as you can and create tests appropriately; and then if there are edge cases/failures in production, you go back and add those to the test suite.

Re: The Servers Are Burning

#27

Post by jschwartzi » Thu, Sep 06, 2018, 7:10 PM UTC

hnruss wrote: post #16

After reading the introduction, I thought that the author was going to use a story of software failure as an example of why you should write unit tests, or at least why you shouldn't deploy untested software. However, the moral of the story was essentially: "software is so complicated that it is bound it break, so you have to be good at fixing it". While that is certainly true, I think that developers have a responsi…

> However, the moral of the story was essentially: "software is so complicated that it is bound it break, so you have to be good at fixing it". While that is certainly true, I think that developers have a responsibility to use whatever tools they can to write high-quality software.

I actually disagree that we have a responsibility to write high-quality software. As engineers our job is to create software that is good enough for the task at hand and that can be improved later.

Sometimes that means that you invest in 100% coverage plus integration and system-level testing because "good enough" means that it has to never fail. Sometimes that means you don't spend much time automating tests and instead do informal development testing before you check something in.

It's entirely up to your judgement based on what your code is going to be used for, and I don't think it was inappropriate for OkCupid's engineering staff to avoid testing in this case. After all, their business was really successful through the period where the error happened.

Re: The Servers Are Burning

#28

Post by mabbo » Thu, Sep 06, 2018, 7:10 PM UTC

andrenotgiant wrote: post #4

JackFr wrote: post #2

Wow. 1. We don't test. 2. We don't code review (or rather if we do, we do it so poorly swallowed exceptions don't raise red flags.) That's an outrageously unprofessional software process.

An outrageously unprofessional software process that created a product worth $50M! https://techcrunch.com/2011/02/02/match-com-acquires-online-...

That's not skill, that's luck. If the bug in question had instead leaked every customer's personal data there wouldn't be a $50m company, there'd be a multi-million dollar lawsuit.

Re: The Servers Are Burning

#29

Post by freekh » Thu, Sep 06, 2018, 7:11 PM UTC

noelwelsh wrote: post #10

How could such a tiny change have such an outsized impact on the site? “That same story happened so many different times,” my old boss David told me. “Someone launched a small, relatively innocuous change that did one of the millions of unexpected things it could have done, which then happened to break some part of the site, and then bring it all down—sometimes bring it down to the point where we couldn’t recover it…

Anecdoctally this happened to me the other day in my Haskell backend: had a recursive function (in the IO monad) that every ten second would take a connection of my connection pool (and since it was recursive) would not give it back to the pool. Of course it was a stupid bug, but it was hard to figure out because it would take a while before it got to that point and once it was there every call to the backend would block then timeout. I like functional programming, and yes: it was in the IO monad so not pure, but I think most applications ends up in some stateful monad even if your stack is built for functional stuff... This was 'not at scale' but it as a Haskell project it is fairly big. Point being: yes, pure apps will not havr this predicament, but even with a stack built for pureness I can see it happen quite often. Running software is messy...

Re: The Servers Are Burning

#30

Post by jschwartzi » Thu, Sep 06, 2018, 7:11 PM UTC

danesparza wrote: post #5

andrenotgiant wrote: post #4

Earlier quoted context omitted.

An outrageously unprofessional software process that created a product worth $50M! https://techcrunch.com/2011/02/02/match-com-acquires-online-...

The OP is about technical process in the software world. I'm not sure how a company's valuation comes into the conversation...

The point is that engineering decisions need to involve both the business and the technical side of things, and they need to be pragmatic and not tied to any particular dogma from either world.