Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update on 1/28 service outage

github.com

Post reply on HN
Print view View original

171–180 of 193 posts

Re: Update on 1/28 service outage

#171

Post by nickpsecurity »

nemothekid wrote: post #116
seiji wrote: post #92

Earlier quoted context omitted.

Exactly right, but over the past six years there's been a strong (and accelerating) trend among developers of " lalala we don't want to know how anything works! give us an API and go away." Most developers I've seen reject even learning about networks or DNS or operating systems or databases. Such willful ignorance boggles the mind, but they are praised because their goals are shipping half-broken things as rapidly a…

> Exactly right, but over the past six years there's been a strong (and accelerating) trend among developers of "lalala we don't want to know how anything works! give us an API and go away." I will argue that this trend has always existed. I'm sure you can find an x86/68k/z80 developer complaining that developers are going "lalala we don't want to know how anything works! give us an the C-language and go away."* I'm…

Or work safely, effectively, and productively when taught how to properly use the abstractions. They can optionally be taught how they work underneath for better results. Yet, I don't have to teach people caches to tell them to group variables closely for performance. I likewise can give very basic explanations of stacks and heaps plus heuristics for using them. People still get the job done.

Functional programming proves my point even more where they don't know how the hardware functions or even use the same model. Yet, with good compiler and language design, they can make robust, fast, and recently parallel programs staying totally within their model. Most problems we pick up outside the abstraction gaps can be fixed in the tooling or with interface checks.

So, I think the common perception of people doing crap code while working within an abstraction is unjustified and even disproven by good practices in that area. Much like I would be unjustified in accusing assembly coders of being "willfully ignorant" or working within foolish abstractions because they didn't know underlying microprogramming or RTL. They don't need it: just knowledge of how to effectively use the assembly. Actually, I saw one commenting so let me go try that real quickly. :)

Re: Update on 1/28 service outage

#172

Post by nickpsecurity »

simula67 wrote: post #144
nemothekid wrote: post #116

Earlier quoted context omitted.

> Exactly right, but over the past six years there's been a strong (and accelerating) trend among developers of "lalala we don't want to know how anything works! give us an API and go away." I will argue that this trend has always existed. I'm sure you can find an x86/68k/z80 developer complaining that developers are going "lalala we don't want to know how anything works! give us an the C-language and go away."* I'm…

As programmers, we need to know atleast one level below the abstractions to which we are programming to. For example, if you program in C you need to know a little bit of assembly, how objects are laid out in memory etc. This is how you write fast code and it helps with debugging too. But if you are programming in C and notices that something goes wrong with the hardware, ( for example, an instruction does not do som…

At least one level below. Hmm. That sounds more reasonable than full understanding. See my reply to nemo, though, for an alternative that I think is more reasonable. Basically, heuristics and simplified models.

Re: Update on 1/28 service outage

#173

Post by nickpsecurity »

semi-extrinsic wrote: post #161
nemothekid wrote: post #116

Earlier quoted context omitted.

> Exactly right, but over the past six years there's been a strong (and accelerating) trend among developers of "lalala we don't want to know how anything works! give us an API and go away." I will argue that this trend has always existed. I'm sure you can find an x86/68k/z80 developer complaining that developers are going "lalala we don't want to know how anything works! give us an the C-language and go away."* I'm…

The good reverend Laphroaig preaches: If the 0day in your familiar pastures dwindles, despair not! Rather, bestir yourself to where programmers are led astray from the sacred Assembly, neither understanding what their programming languages compile to, nor asking to see how their data is stored or transmitted in the true bits of the wire. For those who follow their computation through the layers shall gain 0day and pw…

An assembler elitist with a semi-fallacious argument. Let's rewrite that in view of a lower-level elitist to show it still looks true, shows love for assembler as foolish pride, and still fails to matter in face of good, high-level tools.

If the 0day in your familiar pastures dwindles, despair not! Rather, bestir yourself to where programmers are led astray from the sacred RTL/Transistor language, neither understanding what their assembly languages and microprograms compile to, nor asking to see how their data is stored or transmitted in the true bits of the CPU's network-on-a-chip and memory plus analog values and circuitry many run through at interfaces. For those who follow their computation through the layers shall gain 0day and pwn, and those who say “we trust in our assemblers, our C compilers, our APIs, in our proofs, and in our memory models and ISA models and need not burden ourselves with confusing engineering detail that has no scientific value anyhow” shall surely provide an abundance of 0day and pwnage sufficient for all of us.

Source: LISP, Forth, and Oberon communities who did hardware to microcode to language & OS all integrated & consistent. :P

Re: Update on 1/28 service outage

#174

Post by nickpsecurity »

detaro wrote: post #89
why-el wrote: post #44

Earlier quoted context omitted.

> if you're not regularly cutting power to your data center, you're not building resilience to such a thing happening Would love to read examples on who is doing this and how? Reminds me of Netflix's Choas monkey, only applied to electricity. :p

I remember reading a few years back that Yahoo once a week takes a random data center offline, just to make sure they could do that without issues. They probably didn't actually cut the power ;) But they used it as an argument against investing to much in emergency generators and such: they'll fail or cause accidents and you need the ability to fail-over either way, so make it routine.

I think trying to cut power at least once is better if it's possible. The reason is that digital is just an abstraction over analog, electrical activity. Plus there's actual analog in there doing work, too. So, seeing how all the chips in there respond to an actual and instantaneous drop of the power would be an interesting test of the models they're built against.

Like an above commenter mentioned, weird activity in electrical system can make some products go haywire and even corrupt data in unexpected ways. Of course, simulated takedowns and all appropriate measures for countering common issues should've already happened before a real one. Just to be extra clear there.

Re: Update on 1/28 service outage

#175

Post by nvarsj »

JetSetWilly wrote: post #169
JimDabell wrote: post #163

Earlier quoted context omitted.

> Perhaps a critical app might run on 4 hosts spread across two data centres - you're not going to convince people to have chaos monkey regularly and randomly bringing down these hosts, it would cause real impact and is risky. Yeh in theory it should be able to cope but in reality the scales in most orgs are quite different. The difference between theory and reality is precisely the reason Chaos Monkey and tools like…

That's true, but if an app, say, is running on 4 hosts doing some boutique thing for a small unit of 20 traders, then the practical reality is that they might not want Chaos Monkey bringing down 25% of the throughput randomly, and interrupting whatever actual cash money requests are in progress on a host. Itsa lot easier to promote that if it is thousands of servers doing something fairly mundane where, worst-case, i…

I think you're missing the point. The point of something like chaos monkey is to force you to build a system that won't lose money by "bringing down 25% of the throughput".

Re: Update on 1/28 service outage

#176

Post by nvarsj »

skewart wrote: post #11

Am I the only one who is a little shocked that a power outage could have such a huge effect and bring them down for so long? I'm not an infrastructure guy, and I don't know anything about Github's systems, but aren't data center power outages pretty much exactly the kind of thing you plan for with multi-region failover and whatnot. Is it actually frighteningly easy for kind of to happen despite following best practic…

It is a bit surprising actually. It means they haven't built their app to be tolerant of single DC loss, either on purpose or because they didn't test it properly.

Purely conjecture, but I suspect since github uses mysql cluster they only write to a single dc, which would be the primary dc that failed in this case.

Re: Update on 1/28 service outage

#177

Post by snowwrestler »

nickpsecurity wrote: post #168
snowwrestler wrote: post #164

Earlier quoted context omitted.

Git is very flexible and does not even require repo-to-repo communication over the wire at all; patches can be emailed among contributors and then committed and tracked locally. Branching and merging is so fast and easy in git that every participant can have a slightly different repo for a given project, incorporating shared changes as they see fit. Github is popular because it is opinionated--it chooses to use git i…

That's pretty wild. Sounds like main program/protocol is very true to the UNIX philosophy of tooling. My early reads on it suggested that gave it both it's power/versatility and horrific UI consequences for beginners. An opinionated UI and host like Github is a natural consequence. I remember thinking in my early reading that git was like an assembly language for build systems. It really needed a front-end of some ki…

There have been a number of attempts to build a more friendly front-end for git:

https://git-scm.com/download/gui/linux

Github (which provides a desktop app in addition to their website) is by far the most successful one, I think because they define a whole simplifed and social experience, not just a client.

To me it seems like people seem to segment into two camps: those who want to do the basics (they tend to use GitHub), and those who want to use the full power of git (they tend to use the CLI).

Re: Update on 1/28 service outage

#178

Post by seiji »

nemothekid wrote: post #116
seiji wrote: post #92

Earlier quoted context omitted.

Exactly right, but over the past six years there's been a strong (and accelerating) trend among developers of " lalala we don't want to know how anything works! give us an API and go away." Most developers I've seen reject even learning about networks or DNS or operating systems or databases. Such willful ignorance boggles the mind, but they are praised because their goals are shipping half-broken things as rapidly a…

> Exactly right, but over the past six years there's been a strong (and accelerating) trend among developers of "lalala we don't want to know how anything works! give us an API and go away." I will argue that this trend has always existed. I'm sure you can find an x86/68k/z80 developer complaining that developers are going "lalala we don't want to know how anything works! give us an the C-language and go away."* I'm…

couldn't imagine learning C without learning x86

One difference: C->x86 is a static translation layer. Other network/system things dynamically change out from under your "designed" system and alter threat/security/disaster/reliability/consistency models in a potentially unpredictable combinational fashion.

Saying "cloud abstraction" or "I trust this API and don't care how it works" is basically committing every https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu... and just saying "X can't break because we use provider Y who guarantees they can violate the laws of physics for us!"

Re: Update on 1/28 service outage

#179

Post by kbenson »

forrestthewoods wrote: post #149
kbenson wrote: post #55

Earlier quoted context omitted.

I think that's a bit crazy as well. This is a problem if your build process happens often and requires pulling external data. Ideally, you want a way to cache that external data, and a way to force invalidation of that cache. Building , at least after the first time, should not require external access. There are security reasons for this as well.

So your proposed solution is one of the only two hard problems in computer science? That should be a solid clue that you're wrong. "There are only two hard things in Computer Science: cache invalidation and naming things." -- Phil Karlton

By "a way to force invalidation of that cache" I didn't mean automatic invalidation, I meant a way to flag that you want it to re-download dependencies and store them for later use. I'm not sure where you got the requirement that it needs to automatically determined by a computer from my comment. I was thinking the "cache" could be as simple as the person setting up the build environment downloading the dependencies and configuring the build to use them. That's a local cache, when discussing automatic downloading of dependencies during building.

Set up your build environment with whatever manual intervention is required so that it can run without downloading remote resources. Build as needed. There is no reason for, and many reasons against, downloading dependencies during the build process, but that doesn't necessitate duplicating those dependencies within your own source tree. As long as there are directions on how to download a specific, definitive version of the dependency, whether that is automated or not isn't really a big deal if it's done infrequently.

Re: Update on 1/28 service outage

#180

Post by secure »

ryanfitz wrote: post #83

I recently read a blog post from Github about them operating their own datacenter http://githubengineering.com/githubs-metal-cloud/ Im not positive, but it sounds like a fairly recent switch from a cloud provider to their own datacenter. If thats the case, Id expect a number of outages to come in the following months.

AFAIK, they never used a cloud provider.
Post reply on HN