Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

761–770 of 804 posts

Re: The August 17 outage

#761

Post by menaerus »

lietuvis wrote: post #612
menaerus wrote: post #603

Earlier quoted context omitted.

Hey, traditional hand coder fellow ... the times have past and the future is already a present. I never have been this productive before, and it's been ~20 years that I spent coding. System level programming. Few years back I would have said the bottleneck is not in spelling out the code so we wouldn't see that much AI impact but boy I was wrong. Writing code has never been this cheap, both in terms of time resources…

> I never have been this productive before So, what are you doing/ have done with all your productivity?

Enjoying free time that otherwise I wouldn't have. Building stuff that otherwise I wouldn't have time or resources for.

Re: The August 17 outage

#762

Post by NateEag »

psunavy03 wrote: post #389
hirvi74 wrote: post #382

Earlier quoted context omitted.

I always advocate for the "boomer" editors when possible. It doesn't matter which one you choose. People can say what they want about them, but very little software survives almost 60 years. I sure as Hell hope VS Code doesn't.

At some point, esoteric UIs may have been needed in the 70s, but that ship has long sailed. I don't need to give myself carpal tunnel syndrome trying to quit vim. If anything, a lot of that is also a problem that continues to plague software to this day: devs who go "well if it works for me, it's good enough for anybody, and if you don't like it, you must not be a Real Developer!"

Learning and using vim keybindings was one of the crucial things that mitigated my RSI.

Ergonomics are highly variable, so it might not help someone else, but for me it did the opposite of "give me carpal tunnel syndrome."

Re: The August 17 outage

#763

Post by chrisjj »

Olreich wrote: post #245
Quarrelsome wrote: post #186

Are retries bad? These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic. I feel like I'm mildly stupid…

https://brooker.co.za/blog/2022/02/28/retries.html Seems that retries are good when the error is rare, and bad when the error is common. Typically outages have you transitioning from "everything is fine" to "nothing works", so being able detect that transition early is helpful

> Seems that retries are good when the error is rare, and bad when the error is common.

Retries are a great way to turn errors rare into common.

Re: The August 17 outage

#764

Post by pibaker »

consumer451 wrote: post #106
xyzsparetimexyz wrote: post #52

Earlier quoted context omitted.

wow. they should really institute a maximum amount of individual pushes per-month per-user.

That would just drive users into the arms of the alternatives, which would love to own the world's code... like Cursor/Musk. Microsoft and GitHub's only option is to suck it up, absorb this growth, and lower failure rates. They have the money, so that's not the issue. As someone on the sidelines, this is really interesting to watch unfold.

If you use GitHub as just a software forge then sure you can find an alternative. But I suspect more people use GitHub for its social aspects and they will stick around despite the regressions because building an alternative to an established social network is incredibly difficult if not impossible.

You can get work done on any software forge. But potential employers will still ask for your GitHub. People will judge your personal project by its GitHub stars and be less reluctant to download a binary from GitHub than elsewhere. Potential contributors will leave a PR on GitHub but probably not if they have to make an account on a new platform and learn how it works.

And of course, let's not assume any competitor can just absorb even a fraction of the traffic GitHub receives without suffering similar reliability issues.

Re: The August 17 outage

#765

Post by [deleted] »

Olreich wrote: post #245
Quarrelsome wrote: post #186

Are retries bad? These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic. I feel like I'm mildly stupid…

https://brooker.co.za/blog/2022/02/28/retries.html Seems that retries are good when the error is rare, and bad when the error is common. Typically outages have you transitioning from "everything is fine" to "nothing works", so being able detect that transition early is helpful

[deleted]

Re: The August 17 outage

#766

Post by theamk »

inigyou wrote: post #668
randusername wrote: post #650

Earlier quoted context omitted.

I read the article and found it interesting. It's wild to me that when I think MS I think this article, Vista, Windows Phone, Zune, Metro, ads in the start menu, the pleated pants meme and so-on, but I have coworkers that are total evangelists for MS technologies and platforms. Clearly a lot of devs get value from the ecosystem.

A lot of their platforms (not Azure) used to just work. Visual Studio and .NET were solid. The Windows NT line was pretty good for servers too (but different from Linux, and not as flexible by virtue of not being a bazaar). It was at Windows 8 era they turned it into shit, first by designing a UI for tablets (fine) and then putting that UI on desktops (wtf) and silently upgrading Windows 7 users without their consent…

I remember running IIS and Apache side-by-side many years ago, and oh boy, was IIS a pain.

Instead of a single config dir where you just deploy the configs and you are done, IIS settings were all over the registry. And don't forget component registration - a single, global registry per machine, used by IIS. Want to run staging and dev server on same machine? Tough luck. Random software installs can upgrade components and cause failures in completely unrelated failures...

Re: The August 17 outage

#767

Post by _heimdall »

TorKlingberg wrote: post #749
_heimdall wrote: post #300

Earlier quoted context omitted.

In what way? If I have multiple local commits pushed once I expect to see CI type actions to run once for the push rather than for ever commit in the push.

This is configurable in Github Actions. Many projects want to run CI for each commit to avoid situations where one commit breaks the build and the commit after in the same push or PR fixes it. Broken commits in the history makes bisecting harder.

Interesting, never realized that was an option. Thanks!

Re: The August 17 outage

#768

Post by dilyevsky »

solatic wrote: post #634
dilyevsky wrote: post #605

Earlier quoted context omitted.

Yes, just absolutely crazy way of doing load shedding. > why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway? Like what workloads? Application backends and databases? Did you ever think that your past three employers maybe had a valid point?

The entire GitHub site was unavailable. The "unicorn" page. Total outage. Visible to every user. Worst-case scenario. > Application backends? I don't think I'm taking crazy pills to suggest that it's preferable for services like rendering PR diffs, MR merge trains, even accepting new Git commit pushes, to be temporarily unavailable, so that the entire web application doesn't fall over, and cache-friendly read-only wo…

First of all, web/api being down is not the worst case scenario - i didn't even notice this till late in the day. Secondly, even 1M qps at the edge is tiny amount of compute, what you have here is not a scaling problem it's a misconfiguration problem. Don't make it worse by adding more load on the system with your preemption churn and maybe taking out more important services causing more severe/longer outage - just fix it in the edge.

Re: The August 17 outage

#769

Post by aoeusnth1 »

vasco wrote: post #637
nosefrog wrote: post #599

Earlier quoted context omitted.

I don't understand your comment. A google data center is much larger than an ntp server, but it's obviously not infinitely larger. As you know, if it was infinite capacity, then there would be no need for load balancing or load shedding. And of course, load shedding low priority traffic is still a partial outage, it's just a less bad outage than load shedding high priority traffic. It does not fix lack of capacity, b…

You understand it if you think of engineering infinites rather than mathematical infinites. The capacity of a full single datacenter can be treated as infinite for most customers. I explained how its defined in the original comment. Amount of places in engineering where you treat even a 3 order of magnitude difference as infinite is a lot, but the number of order of magnitudes varies depending on context.

No customer has access to an appreciable fraction of any data center, and if they do they don't have 10x overhead in reserved quota.
Post reply on HN