Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

421–430 of 804 posts

Re: The August 17 outage

#421

Post by antonvs »

busterarm wrote: post #305
TomBombadildoze wrote: post #250

Earlier quoted context omitted.

[flagged]

I solve real problems at scale and the engineering practices I see on a daily basis are a clown show. There's little to no basic understanding of networking, distributed systems, databases, etc. 99% of our engineers were hired from their college internships and never worked anywhere else. Industry hires to improve systems rarely last more than a year and it is almost never their fault. We're in the next tier down fro…

I had to explain to a “senior” engineer the other day (read: a few years experience) why locating a database client in a different geo region from the server is a bad idea (especially when that client is using an ORM that likes to make lots of little calls to the server.)

I also had to argue for changing a system that was reading about 100k small files from cloud storage to use a single compressed file. There seemed to be no awareness that copying 100k files might be inefficient.

Re: The August 17 outage

#422

Post by tcmart14 »

altcognito wrote: post #265

Distributing across different services wouldn't be a bad idea.... I still can't help but feel a little grateful for what they do across the free side of things. I know it isn't altruism, and I know nobody needs to defend a billion dollar corporation but... Name another service that does what they do for FREE (and no ads) at this scale. It isn't easy. Wikipedia has probably more usage, but is a simpler endeavor. (exce…

My only push back would be on the FREE part. I coulda bought that 5 years ago. Now I look at GitHub and go, "if the product is free, it's because Im the product" with all their co-pilot stuff.

Re: The August 17 outage

#424

Post by silver92bullet »

This article seems to say that there is "no excuse" for these issues but look at all these things we changed and are changing. It doesn't really feel transparent and it feels like they aren't really taking true ownership on what has happened.

Re: The August 17 outage

#425

Post by lbrandy »

unscaled wrote: post #403
Breakthrough wrote: post #357

Earlier quoted context omitted.

You have my empathy for this kind of sentiment. Personally this seems somewhat rare in practice. That being said I'm curious if anyone has anecdotes they can share about these kinds of things?

I think the "happy path" might be a slightly wrong classification in GP, since the post is in reply to a retry-storm issue and explicitly talks about retry storms and thundering herds. I've seen many cases where engineers optimize the sad path, but pessimize the wretched path. Or in less flowery language, they cut the occurrence rate of common non-critical failures, but by doing that they introduce code that can make…

> I've seen many cases where engineers optimize the sad path, but pessimize the wretched path.

As said GP, yes this is exactly what I meant and "happy-path" was an unclear choice of words. They optimize for the unhappy path on the good days, and make the bad days much worse.

Re: The August 17 outage

#426

Post by dang »

ACCount37 wrote: post #208
cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

"Please don't sneer, including at the rest of the community." It's reliably a marker of bad comments and worse threads, and you can make your substantive points without it.

https://news.ycombinator.com/newsguidelines.html

Re: The August 17 outage

#427

Post by wat10000 »

frollogaston wrote: post #374
wat10000 wrote: post #262

Earlier quoted context omitted.

I have the exact opposite view. Way too often, I’ll be presented with an error to the effect of, “something went wrong, please try again” and often the retry works. And I’m left wondering why this machine whose sole purpose is to automate things can’t do that for me automatically. In particular, networks tend to be a LOT less reliable than the typical developer accounts for. And the failures are very often transient.…

The Github outage was about internal clients. Phone apps are a reasonable place to say things are known to be unreliable and can't be fixed. Your IP address changes when you leave the house. Btw, PWAs added offline capabilities to websites. I hate how the only thing that got used for was these stupid pages that look like you were able to reach the site but it's actually just saying you have no internet, like YouTube.

I’m sure some retries are helpful there too (TCP is doing them, at the very least) but yeah, different approaches for different situations. Maybe you retry but you don’t spend many seconds hoping for it to work.

Re: The August 17 outage

#428

Post by antonvs »

jambalaya8 wrote: post #337
ACCount37 wrote: post #208

Earlier quoted context omitted.

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

I think a lot have mostly experienced working for tech companies and do not understand how different that is from working at some other kind of company (particularly something not technically sexy) doing tech (see last parenthetical).

At a tech company, engineers mostly work on software that is the company’s product, and as such its functionality, reliability etc. are high company priorities, and the whole company is focused on producing and selling those products.

At a non-tech-company, that’s not the case. Which means you end up with something much closer to the movie “Office Space”, where software is just considered a necessary evil that’s worked on by what’s essentially the software development arm of an IT department, where any opportunity to cut costs or headcount is considered a good idea, where “best practices” take a back seat to “any practice that Bob can make work.”

A lot of companies these days are somewhere between the two: their main product is not software, but the company depends on a website to interact with a large number of customers: banks, insurance companies, web stores, etc. In that case, you tend to get the worst of both worlds, naturally.

Re: The August 17 outage

#429

Post by msephton »

jryan49 wrote: post #336

With all the software being written on github you'd think we were going though a software rennasance. Where are the results? Is it really just all slop?

I can't speak for all of it, obviously, I don't have time to try much of it, but I see tons of amazing new software in my various feeds pretty much daily.

Re: The August 17 outage

#430

Post by mrheosuper »

eszed wrote: post #261
danudey wrote: post #248

Earlier quoted context omitted.

AI finding issues in code and reporting them so that an AI can review and triage them for another AI to fix.

I mean, isn't that the dream? I don't know if that's sarcasm or not. I know it doesn't work , but that's the future we've been promised, right?

Let's just say my dream does not involve babysitting some robot
Post reply on HN