Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

611–620 of 804 posts

Re: The August 17 outage

#611

Post by concerned_user »

xnorswap wrote: post #606
d0vs wrote: post #561

Earlier quoted context omitted.

GH didn't collapse so it's not impossible that they already implement these measures: > At peak, web/API error rates were approximately 20%, while archive and raw-content downloads reached approximately 50%. https://www.githubstatus.com/incidents/zkxwbgr0cnmx

It completely collapsed, 20% is meaningless, as I said in: https://news.ycombinator.com/item?id=49333089 If you load an issue page, you'll see 1 failed request to: /project/product/issues/ And sure, that's what you care about, but consider the working requests to: /in-product-messaging/copilot-budget-request-banner /in-product-messaging/code-scanning-ai-findings-preview-banner /github-copilot/chat /_private/browser/s…

If page has 15 requests and needs data from all of them to work correctly, then with 20% failure rate you are suddenly close to 100% non-functional page from the user perspective.

Re: The August 17 outage

#612

Post by lietuvis »

menaerus wrote: post #603
sph wrote: post #559

Earlier quoted context omitted.

It's starting to become a cliché to have people reply "I'm getting a lot more done", but without seeing any evidence of this incredible productivity gains, I'm starting to wonder if y'all are suffering from collective hallucination. If the accepted claims are of "100x productivity" (increasing by the day), and LLMs have gotten very good for the past ~year, for sake of argument, where are the 100 year improvements in…

Hey, traditional hand coder fellow ... the times have past and the future is already a present. I never have been this productive before, and it's been ~20 years that I spent coding. System level programming. Few years back I would have said the bottleneck is not in spelling out the code so we wouldn't see that much AI impact but boy I was wrong. Writing code has never been this cheap, both in terms of time resources…

> I never have been this productive before

So, what are you doing/ have done with all your productivity?

Re: The August 17 outage

#613

Post by tremon »

charrondev wrote: post #346
jeremyjh wrote: post #299

Earlier quoted context omitted.

None of the things you mention - indexing or secret scan would be done individually for each commit. As I already said, this would be a log of all commits in the range pushed - it would be scanned once for those things. There is no need for a loop running over a range of commits and processing each one.

There 100% is at least for things like secret scanning and message parsing. Secret scanning needs to make sure my repo as a whole has no secrets. It’s not acceptable to have 1 commit introducing it and 1 removing it because the secret is still recoverable. Every commit is also surely an entry in a database somewhere. I can navigate in GitHub directly to any individual commit so there is definitely some overhead of so…

It's not necessary to perform secret scanning on a per-commit basis. The most efficient way is to just scan all blob objects being pushed; there is no reason to even be aware of an object's location (tree path or commit) except for diagnostic messages.

> I can navigate in GitHub directly to any individual commit

You can do the same with the git command line client. The overhead you claim is already in the git on-disk format. Github might very well duplicate this information in a database somewhere, but it doesn't follow from your observation.

Re: The August 17 outage

#614

Post by mrweasel »

aesthetics1 wrote: post #189

> Since April, monthly commits have grown from 1.4 billion to 2.9 billion Bonkers. You can tell the entire industry is in a "productivity panic" and here's more proof. There's a velocity zealot crying tears of joy somewhere.

Crazy how more code haven't really yielded better products. Who would have guess that the number of lines of code wasn't the problem. Except that very basic tools that I use daily, pretty much every piece of software has become worse in the past few years. The closer to the consumer the software sits, the worse the quality is becoming.

Re: The August 17 outage

#615

Post by XzAeRosho »

TiredOfLife wrote: post #608
neya wrote: post #602

Earlier quoted context omitted.

Let's be honest here, the real reason is the crap that is Azure. GitHub was perfectly fine until then. They are just too bureaucratic to admit it. Worth reading: https://isolveproblems.substack.com/p/how-microsoft-vaporize...

So for the past 6 years the bad uptime was because of less than 12% of github being on azure?

Read the article. It's about Microsoft culture messing things up, not Azure themselves (though it may be a factor).

Re: The August 17 outage

#616

Post by catlifeonmars »

Brian_K_White wrote: post #604
nosefrog wrote: post #599

Earlier quoted context omitted.

I don't understand your comment. A google data center is much larger than an ntp server, but it's obviously not infinitely larger. As you know, if it was infinite capacity, then there would be no need for load balancing or load shedding. And of course, load shedding low priority traffic is still a partial outage, it's just a less bad outage than load shedding high priority traffic. It does not fix lack of capacity, b…

It is infinitely larger, because there is no distinction between more than you use, and infinity.

Sure there is infinite capacity given an infinite amount of time to scale up. I assert you’re leaving out the time dimension. Those so-called infinities are simply not accessible in a practical way since you’ll hit a wall in actually provisioning that capacity long before the data center runs out of compute.

Re: The August 17 outage

#617

Post by ramon156 »

hnburnsy wrote: post #396

>We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity. We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure. Crazy.

they're still migrating to Azure?

Re: The August 17 outage

#618

Post by juggle-anyhow »

dca88 wrote: post #440
peab wrote: post #414

Earlier quoted context omitted.

This comment comes up over and over again and it's incredibly ignorant. To give just a single example, ai code dev has enabled people to make tools for themselves that they didn't have before. I've made a language learning app for myself. Its working better than Duolingo so far, for me. Its not really public

Why is it incredibly ignorant? Where is he wrong? We have definitely lost the plot.

and where is your evidence of that?

Re: The August 17 outage

#619

Post by m4rtink »

sandos wrote: post #568
ethagnawl wrote: post #386

Earlier quoted context omitted.

Bonkers is right. Where in those ~12 billion commits is the software, products and "innovations" which are supposed to be making our lives better? Software and apps in particular are getting worse, normies hate AI more than ever because they're even less likely to get their desired outcome when calling their doctor or trying to get their online order refunded when chatting with a cutely named chatbot, wages for (most…

As a hobby I am playing around with mesh radios using Lora. Let me tell you, it is _silly_ the amount of vibe-coded software in this area is popping up every week. Is the software duplicated? To an extremely large extent, yes. It is useful? Yes, but each piece of software seems to have a smaller and smaller audience, and quality is often severely lacking. Iv'e done my own, too, for "RF debugging" as I called it to lo…

Yeah, a lot of the software around MeshCore people have created seems heavily vibe coded (since got used to having press a button multiple times until it works) & the community seems to be surprisingly non-critical to "AI" technologies. Is it just about timing or something else ?

Re: The August 17 outage

#620

Post by draw_down »

d0vs wrote: post #561
afc wrote: post #531

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling fai…

GH didn't collapse so it's not impossible that they already implement these measures: > At peak, web/API error rates were approximately 20%, while archive and raw-content downloads reached approximately 50%. https://www.githubstatus.com/incidents/zkxwbgr0cnmx

Shyeah, all those successful requests were really helpful for showing me a page, which then failed when I tried to do anything by clicking the buttons on the page. I was completely unable to merge things for hours, for no lack of trying. GH was down.
Post reply on HN