Earlier quoted context omitted.
"exponential" and "slightly longer" are very different backoff patterns.
The typical ceiling for these is around a minute.
Which is probably hundreds of times your normal wait.
The August 17 outage
771–780 of 804 posts
Re: The August 17 outage
#772Earlier quoted context omitted.
> Where in those ~12 billion commits is the software, products and "innovations" which are supposed to be making our lives better? Where is anything saying there should be? If they were all private hobby projects, you'd have no grounds for complaint.
> Where is anything saying there should be? What are the boosters ranting and raving about if not the above? Am I just confused and the end game for all of the AI hype, investment, infrastructure build out, etc. over the last N years is ... hobby projects? Here's a quote I found after a quick web search for "dario ai benefits": > The list of positive applications of powerful AI is extremely long (and includes robotic…
It's hard to separate out the hype from the results, but if you think none of that's being tried or worked on then you're the Ostrich with your head in the sand. Mayo Clinic says AI models help detect pancreatic cancer on CT scans 3 years before clinical diagnosis.[1] Breast Cancer.org says emerging research shows AI might spot cancers in mammograms that radiologists might miss[2]. Nature.com has a paper conclusing that AI (in various forms) have "already revolutionised the diagnosis, prognosis, screening, biomarker discovery" of cancers.[3] Cancer.gov says NCI scientists are using AI to improve cervical cancer screening with deep learning and digital images[4].
Dr David Burns offers his Teams CBT therapy with an LLM based app for people who can't afford therapist visits[5].
U. Chicago says that AI weather forecasts can be faster and cheaper than supercomputer-based physical simulations and produce results weeks ahead, which makes accurate forecasts affordable and available for farmers in middle-income countries[6].
With your smartphone camera you can get an AI-assisted bike-fit at home[7].
You can get a bike fitness training plan[8] and performance analysis with an AI that you can chat with to analyse and explain or change it, used by three professional cycling teams.
Prophix Australia analyse photos of powerlines taken by helicopter using AI to track assets and identify and verify potential faults[9]. AI systems scan images for early signs of plant disease in the vineyards of the Swiss Alps [also 9]. AI systems analyze multispectral images and satellite images of crops to detect nutrient deficiencies or pest problems in large farms[also 9].
In 2020 - 2022 a trial of a system in a hospital to identify patients at risk of sepsis helped medics treat them 2 hours earlier and reduced the death rate from sepsis by 18%.[10]
Self-driving cars now exist. Camera drones which follow you and respond to gestures now exist.
[1] https://newsnetwork.mayoclinic.org/discussion/mayo-clinic-ai...
[2] https://www.breastcancer.org/screening-testing/artificial-in...
[3] https://www.nature.com/articles/s41698-026-01276-6#Sec17
[4] https://www.cancer.gov/research/infrastructure/artificial-in...
[5] https://www.feelinggoodapp.com/
[6] https://climate.uchicago.edu/insights/ai-is-transforming-wea...
[7] https://www.myvelofit.com/
[9] https://medium.com/design-bootcamp/what-has-ai-ever-done-for...
[10] https://www.scientificamerican.com/article/algorithm-that-de...
Re: The August 17 outage
#773Earlier quoted context omitted.
Just write the tests. It's your job, not your employer's.
It's their code, not mine. If they don't want us writing tests, then so be it. I get paid either way.
You're the engineer, not them - you're being paid to write code that ideally doesn't break - it's your job to explain that tests benefit them. There's very little excuse not to add test coverage as you go along these days given an LLM could help write most in a flash.
Or, don't tell them and do it anyway. It'll make your own life 100x easier, unless you enjoy the stress of bug fixing and responding to incidents where you've broken something.
Writing code without tests because it takes too long is like running a car engine without ever bothering to change the oil - it'll run for a bit, and eventually fail catastrophically.
Re: The August 17 outage
#774This seems like a pretty straightforward and easily winnable situation for GitHub. The demand for their services just doubled, apparently. They have no real competitor operating at the scale they’re at. They have pretty substantial network effects.
They are under no obligation to continue functioning as a bottomless free repository for text file hosting, especially now that text file creation has multiplied exponentially. They could make a few almost purely commercial changes and solve this without any major re-engineering while maintaining their status as the go-to public / open source code hosting platform.
1. Immediately increase pricing of all enterprise licenses and add super-committer overage fees.
2. Rate limit or cap commit size / frequency for public accounts.
Their service is more valuable than ever and switching is much harder if people have automation built up on their platform. Now is the time to cash in their chips.
And the positive externality of increasing commit cost would be forcing people to have some semblance of restraint for the AI content they generate.
Re: The August 17 outage
#775Earlier quoted context omitted.
The typical ceiling for these is around a minute.
This can either be a relatively short time or an eternity, depending on the upstream consequences. If that means holding a connection open for 60000 milliseconds while waiting for some downstream rpc to go through its backoff ritual, that's an eternity, and under load that connection pool will get exhausted quickly. So now a problem which should only affect maybe 1% of users has completely hosed everyone. I've seen t…
We are probably talking past each other.
I'm not proposing doing any exponential backoff retries, or even retries at all for internal services.
In my mind, the retries with exponential backoff and jitter belong only on the end-client(VSCode in this case). Everything else -> fail fast.
Re: The August 17 outage
#776Earlier quoted context omitted.
This can either be a relatively short time or an eternity, depending on the upstream consequences. If that means holding a connection open for 60000 milliseconds while waiting for some downstream rpc to go through its backoff ritual, that's an eternity, and under load that connection pool will get exhausted quickly. So now a problem which should only affect maybe 1% of users has completely hosed everyone. I've seen t…
We are probably talking past each other. I'm not proposing doing any exponential backoff retries, or even retries at all for internal services. In my mind, the retries with exponential backoff and jitter belong only on the end-client(VSCode in this case). Everything else -> fail fast.
With http you can do it elegantly with a 429 and retry-after. 100% doing it at the edge is the way to go. The machinery which determines how long to delay a client's retry can benefit from knowledge of internal services' state, but I agree the ultimate decision must lie with the serving layer. That's the only way to efficiently deal with misbehaving clients.
On that note, one of the more memorable incidents of my career was when a 10M+ node client decided to retry as hard as possible on 4xx. That was fun x_x.
[edit] that is to say, for this mechanism to be robust your retry-after enforcement mechanism needs to be capable of withstanding almost every single one of your users attempting to illegally retry as fast as they physically can without negatively impacting that one user requesting legitimate traffic. https://media.tenor.com/p3mss3YI6TcAAAAM/wat.gif
Re: The August 17 outage
#777Earlier quoted context omitted.
usual crying from the usual people. AI slop, blahblahb, ... (I don't mean you. just these so called open source developers.)
I'm not sure if it's going to persuade you but here is an example: https://github.com/uclouvain/openjpeg Basically the only library for reading jp2k data (complicated specs, ask your AI to one shot an implementation, mine said "it's 3000 lines of fiddly spec, too complicated"). Issues full of buffer-overflows. Recently unmaintained. Used in tons of projects, now all possibly vulnerable.
and? don't use it. switch. abandon.
did we lose anything of value? probably not.
Re: The August 17 outage
#778Earlier quoted context omitted.
This comment comes up over and over again and it's incredibly ignorant. To give just a single example, ai code dev has enabled people to make tools for themselves that they didn't have before. I've made a language learning app for myself. Its working better than Duolingo so far, for me. Its not really public
> ai code dev has enabled people to make tools for themselves that they didn't have before. ...that sounds pretty worthless tbh. The promise of AI wasn't that you can bang out your own little half baked duolingo clone. It was that every engineer was supposed to become hundreds or thousands of percent more productive. The assumption being that things generally would therefore get noticeably better . We're seeing hundr…
Out of curiosity, what real economic terms would you have to see to be convinced?
Do you mean more earnings per share for corporations or real utility impacts on social systems (e.g., more new drugs)
When I ask Claude about the AI ROI, it seems to cite that ~95% of AI PoC's are negative ROI but the 5% that do have ROI tend to have decent return -- my guess is mostly back office clean up to reduce expenses and increase revenue/profit.
IIRC Uber claim their 1.5k/mo budget lead to no new value creation.
FWIW, I look at the situation with similar skepticism. One argument you could make was what Marc Andreesen said with the hypothesis that "all the big companies get nuked", so no new valued gets created but the large market cap companies get eaten by thousands of little pirañas. Obviously, so far, that doesn't seem to be the case.
Re: The August 17 outage
#779Earlier quoted context omitted.
Google “thundering herd” and you’ll understand why uncontrolled retries can be / are bad.
Quarrelsome is suggesting you should have no retries, not unlimited.
Yes I’m aware, and they’re probably suggesting that something on a higher level should instead retry, and I’m arguing that you need to coordinate the retry mechanisms and behavior between differently layers of your stack.
Re: The August 17 outage
#780Earlier quoted context omitted.
Absolute numbers aren’t useful in this analysis. We know capital is easily available for the expenditure. You have to consider it as a relative number to revenue or users otherwise big numbers always look scary.
I agree - however, I was responding to your direct question about whether GitHub's operating cost was high, and there's some data that suggests that it is!
If this is one of their major expenses and only represents 5% of annual revenue, it’s not. Often SaaS companies can run at 90%+ gross margins even with a highly utilized free user tier. Their selling expenses and labor/opex and SG&A run rates can be all over the map but these are extremely controllable and represent the operating strategy being executed more than the cost of service.