Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

551–560 of 804 posts

Re: The August 17 outage

#551

Post by okeuro49 »

fsloth wrote: post #520
yread wrote: post #512

Earlier quoted context omitted.

Other projects get unmaintained with maintainers burnt out by a torrent of vulberability reports

This I don't understand. If it's not your job, then just ignore the reports. If it's actually critical, someone will put money on the table and then it's a business. And then it's about scheduling and resourcing - also should not burn anyone out. Just because many people have false sense of entitlement as soon as they get a free offering, it does not mean anyone needs to accommodate them.

> If it's not your job, then just ignore the reports.

If you have a highly conscientious personality, this is easier said than done.

Re: The August 17 outage

#552

Post by user43928 »

bch wrote: post #532
bdlowery wrote: post #503

Earlier quoted context omitted.

you seem to be uninformed. This should help: https://www.youtube.com/watch?v=H_c6MWk7PQc

> raised heat is a kind of pollution > water is not unlimited > you don't want to release a bunch of acid into a river > I know next to nothing about this > I know basically nothing > corn is one of the thirstiest major crops grown in the US What is this rant supposed to inform? Whats wrong with OP being concerned about the costs of operating a DC?

Nobody is concerned about the costs and environmental impact of data centers.

When someone in a discussion about the benefits of AI goes "did you think about the environment?!", it's always performative.

The real motivation is disliking AI itself or doubt about the government's ability to offset the labor market impact. Discussions that start with feigned concerns being raised are nearly always going to be unproductive.

Re: The August 17 outage

#553

Post by eddieh »

bdlowery wrote: post #503
croes wrote: post #486

Earlier quoted context omitted.

Are they happy about the circumstances of AI too? Higher resource consumption and the set back in CO2 reduction?

you seem to be uninformed. This should help: https://www.youtube.com/watch?v=H_c6MWk7PQc

A link to a YouTube video without context e.g. summary of findings, primary author/creators, and primary citations, methodology, and so on is a next to useless for making a point. For all we know you’re linking to a crackpot or an industry sock puppet and I don’t care to “watch” any of what can potentially be a dubious video or worse a malicious video. It is your job as the linker to convince me the video is worth even one iota of my time.

On the other hand recent papers highlight the validity of concern/suspicion:

> This systematic review demonstrates that the environmental footprint of artificial intelligence is a structural and increasingly consequential challenge, shaped by interdependent decisions across algorithms, software pipelines, hardware infrastructures, and deployment contexts. The synthesized evidence shows that energy consumption and carbon emissions associated with AI systems are highly variable, context-dependent, and often underestimated — Beyond Efficiency: A Systematic Review of Energy Consumption and Carbon Footprint Across the AI Lifecycle (published in “Sustainability” an international, peer-reviewed, open-access journal) https://www.mdpi.com/2071-1050/18/3/1359

Re: The August 17 outage

#554

Post by eckesicle »

afc wrote: post #531

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling fai…

This is an excellent book and its lessons saved my bacon many times!

As it happens I have a hardcopy of this book (along with "Seeking SRE" and the "SRE Workbook") that I am giving away (because of a move). If you want a hardcopy then email me your UK address I will be happy to post them to your for free.

I tried putting them on the street in a little box but surprisingly none of my neighbours grabbed any of my software books. :)

EDIT: The books have been given away

Re: The August 17 outage

#555

Post by aryamccarthy »

msephton wrote: post #429
jryan49 wrote: post #336

With all the software being written on github you'd think we were going though a software rennasance. Where are the results? Is it really just all slop?

I can't speak for all of it, obviously, I don't have time to try much of it, but I see tons of amazing new software in my various feeds pretty much daily.

I'm a bit weary of seeing comments like this, but I want to believe you, so help me learn? What are some examples of the amazing software you've seen recently?

Re: The August 17 outage

#556

Post by user43928 »

mikae1 wrote: post #479
gabriel-uribe wrote: post #473

Earlier quoted context omitted.

More done + more leisure.

That's how it's sold, but have you heard about any major tech company that sends their worker home when they've achieved what they used to achieve pre-AI? EDIT: Instead of simply down-voting, you're welcome to name examples that proves me wrong ;)

The only way that you can translate productivity gains into more free time in a salaried position is by working remotely. Seems obvious, no?

Re: The August 17 outage

#558

Post by solatic »

afc wrote: post #531

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling fai…

My last three employers refused to take advantage of Kubernetes PriorityClasses and agree to schedule work to agree (as a cluster-wide resource that affected many teams) on what our PriorityClasses should be and to migrate workloads to have priorities. And this is something relatively easy to implement - no developer work required, and practically no YAML to write.

Why not? Because sadly, fundamentally, most workplaces are not run by people who care about day-2 operations or long-term health. Product or Sales pushes customer-visible work into the pipeline, and you dare not say no. "Day-2" work is not considered to be something that moves the needle. Even now, with GitHub facing these severe outages, it's not like they're facing some massive exodus; their load seems to be getting worse over time, not better.

I'd be very surprised if there weren't any employees at GitHub who had read the SRE book. I'd expect that they're just not listened to.

Re: The August 17 outage

#559

Post by sph »

brookst wrote: post #462
m4rtink wrote: post #417

Earlier quoted context omitted.

So much more stuff and growing- what it is actually useful for ? Are we getting actually more done than with previous volumes or is it just all wasted energy?

I’m getting a lot more done. Hobby projects that languished for years are coming along great, at quality and depth I could never have found time for before.

It's starting to become a cliché to have people reply "I'm getting a lot more done", but without seeing any evidence of this incredible productivity gains, I'm starting to wonder if y'all are suffering from collective hallucination. If the accepted claims are of "100x productivity" (increasing by the day), and LLMs have gotten very good for the past ~year, for sake of argument, where are the 100 year improvements in the status quo of software?

If one claims such extraordinary figures of 100x increased productivity, a step forward never seen in the history of humanity in such short timespans, they must present extraordinary proof or be branded as a complete lunatic. I could have accepted people saying "I'm 20% more productive", which is an incredible achievement by itself, but not the 10x, 20x, 100x I keep hearing about. I think I've read 200x this week.

Re: The August 17 outage

#560

Post by vasco »

afc wrote: post #531

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling fai…

Of course there's infinite capacity. A google datacenter is infinite capacity from the perspective of say an NTP server. Infinites exist when you have enough orders of magnitude in the middle.

Also traffic isolation and degradation by tier is not "no outage", you're still in outage land, you're just being smart in how you use it and choosing what you disrupt. It doesn't fix the lack of capacity.

Post reply on HN