Earlier quoted context omitted.
It's their code, not mine. If they don't want us writing tests, then so be it. I get paid either way.
You're the engineer, not them - you're being paid to write code that ideally doesn't break - it's your job to explain that tests benefit them. There's very little excuse not to add test coverage as you go along these days given an LLM could help write most in a flash. Or, don't tell them and do it anyway. It'll make your own life 100x easier, unless you enjoy the stress of bug fixing and responding to incidents where…
The August 17 outage
791–800 of 804 posts
Re: The August 17 outage
#792Earlier quoted context omitted.
So, I lived through the Internet Explorer age so I'm familiar with that. I think it is a leap to say there are better alternatives dying to get out. There's no objective way to decide they are stifling innovation. I'm not sure there's some solution for what you're complaining about other than the government coming in and saying they have to split github. What are you advocating for here, or is this just complaining?
Why do you think it has to be one of those?
Re: The August 17 outage
#793Earlier quoted context omitted.
Why do you think it has to be one of those?
I’m not advocating for anything (I was just spitballing) I’d be more interested in what you think would be the legal and fair thing to do.
Re: The August 17 outage
#794Earlier quoted context omitted.
Here's the most recent nontrivial one as of this comment: https://github.com/oven-sh/bun/commit/d4de65e9a43224a14591ad... The code change makes no sense and should do nothing. The commit message described a very deep investigation into garbage collection on the C++ side. Some object is being kept alive when the test requires it to be collected, and changing the code in this way allegedly prevents that. But wouldn't y…
The first is, annoyingly, a relatively common problem and solution when dealing with GC lifetimes in tests. Few interpreters/JITs want to generate extra instructions to null out stack slots or pre clobber registers to ensure something becomes collectible at a specific point. Eager nulling of a variable often gets removed by dead store elimination or even just from being a disconnected SSA node. I've written extra nes…
Re: The August 17 outage
#795Earlier quoted context omitted.
> You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour Not to join the parade, but what would a unit test that confirms a cycling behavior across all the instances in-flight even look like? I mean, besides "Not a unit test".
Mock out the network call (which you'd be doing anyway because unit tests never connect to the network) so it always returns a failure. Mock out the timer. Call the function multiple times and ensure it'd passing the expected wait durations in for each time it's called followed by a fatal error after say 30 seconds.
Man, the day this hits, just know I didn't judge you at all.
Re: The August 17 outage
#796Earlier quoted context omitted.
The first is, annoyingly, a relatively common problem and solution when dealing with GC lifetimes in tests. Few interpreters/JITs want to generate extra instructions to null out stack slots or pre clobber registers to ensure something becomes collectible at a specific point. Eager nulling of a variable often gets removed by dead store elimination or even just from being a disconnected SSA node. I've written extra nes…
If this is needed in tests it needs to be known how it works, it needs to work consistently, and it needs to be documented how it works. It can't be an ad-hoc deep investigation and random fix each time. The comment should then be just // ensure foo is no longer a GC root, see gc_roots.md
Re: The August 17 outage
#797Earlier quoted context omitted.
its probably happened on half the core teams, idk what you want me to say, it sounds like you dont have alot of experience
Instagram, Facebook and even threads all had much more mundane growth rates and definitely no unexpected jumps like GitHub is experiencing. I'm sure if suddenly the solar system had 10 more earths with each about 10 billion people and they would all start using Instagram tomorrow we would have exactly the same growing pains and outages that GitHub has today. Luckily for Meta agents are not yet as much into doomscroll…
Re: The August 17 outage
#798Earlier quoted context omitted.
If this is one of their major expenses and only represents 5% of annual revenue, it’s not. Often SaaS companies can run at 90%+ gross margins even with a highly utilized free user tier. Their selling expenses and labor/opex and SG&A run rates can be all over the map but these are extremely controllable and represent the operating strategy being executed more than the cost of service.
Would you say that the operating cost of running an airline is high or low?
I say that because every thing that goes into the service is expensive. Large capital expenses (planes, probably a lot I don’t even know about), large variable operating costs (fuel, flight specific labor, airport fees, and all the other stuff that is required for every flight). A large portion of their expenses scale with revenue. Meanwhile, there is high revenue pressure (customers are price sensitive and competition is fierce and often a direct substitute).
SaaS is inherently more scalable than practically any other business/industry. Comparing to an industry that is exceedingly financially constrained, often to the point of requiring government subsidies, is frankly a bit silly if you’re trying to make any point here.
Let’s also look at it from a risk of growth perspective to highlight how different things are. If GitHub invests in a new server/rack/datacenter to support growth it likely breaks even once it’s at However, what does an airline have to do to rollout service to a new city? They need to buy new planes, sign contracts with the new airport, hire pilots and crew, launch a marketing campaign informing customers to generate demand, and probably some other sunk costs. Once service begins, they want to be at full capacity immediately. They can’t power down 80% of fuel or labor. It will burn cash until they reach a much higher utilization rate. Closing the city is a major public failure for the airline that will be discussed in media, among customers, and could easily lose people their jobs. It’s very risky in comparison is my point.
Re: The August 17 outage
#799Earlier quoted context omitted.
Would you say that the operating cost of running an airline is high or low?
I don’t know that industry as well. But I can look at it through a similar lens and would guess that it’s high. I say that because every thing that goes into the service is expensive. Large capital expenses (planes, probably a lot I don’t even know about), large variable operating costs (fuel, flight specific labor, airport fees, and all the other stuff that is required for every flight). A large portion of their exp…
My point is very simple: operating costs are relative. But you didn't ask the relative question, you asked the absolute question ("what suggests GitHub has a huge operating cost?"). Reasonable people will differ on how to evaluate the answer to that. For example, to a wealthy person, $100 might not be a lot of money. To a poor person, that is a lot of money.
You yourself said: “Absolute numbers aren’t useful in this analysis.”
So perhaps you meant to ask a different question all along.
Re: The August 17 outage
#800Earlier quoted context omitted.
It used to be, and buying a few servers was a cheap way to get off the cloud (if you can manage them efficiently). But with RAM and other costs these days... I've had to rethink it a lot.
It's at worst ten times as expensive as it should be. But AWS was already ten to a hundred times as expensive, so even in the computer parts crisis, self hosting is still ahead.