71–80 of 88 posts
Unfortunately right when we were trying to deploy a hot-fix to production, our CI can't clone the PR to run tests. What do other folks use to avoid this situation? Have a Gitlab instance or similar that you can pull from instead for CI?
> What do other folks use to avoid this situation? Don't use Microsoft?
Earlier quoted context omitted.
You might be surprised to learn that there are many teams responsible for many things, and SRE is very likely a team different than the one that handles user experience decisions
They should have went to the mat over that ridiculous, pc, woke bullshit decision.
Obviously you're entitled to your opinion and welcome to share it, but don't be surprised when that begins having repercussions on your career, and don't be surprised when people on this forum in particular don't respond well to it.
HN is focused on technology and business, not so much politics or your personal opposition to making historically oppressed individuals feel more comfortable in tech, a traditionally white & male dominated space. If you're going to make reasoned arguments around why changing the name of a branch has a demonstrable negative impact on your life, please do. If not, this isn't the place to air your grievances with crass and unproductive comments.
Unfortunately right when we were trying to deploy a hot-fix to production, our CI can't clone the PR to run tests. What do other folks use to avoid this situation? Have a Gitlab instance or similar that you can pull from instead for CI?
Well unfortunately it seems everything I said 11 days ago has become a reality I'm afraid and I was still downvoted for pointing this truth out. [0] Too many times I suggested everyone to begin self-hosting or have that as a backup but once again some think 'going all in on GitHub' is worth it. (It really is not the case) [0] https://news.ycombinator.com/item?id=26301750
Hi all - I'm the head of engineering at GitHub. Please accept my sincere apology for this downtime. The cause was a bad deploy (a db migration that changed an index). We were able to revert in about 30 minutes. This is slower than we'd like, and we'll be doing a full RCA of this outage. For those who are interested, on the first Wednesday of each month, I write a blog post on our availability. Most recent one is here…
Any comment or insight you can share on the overall increase in downtime over the past few years?
Hi all - I'm the head of engineering at GitHub. Please accept my sincere apology for this downtime. The cause was a bad deploy (a db migration that changed an index). We were able to revert in about 30 minutes. This is slower than we'd like, and we'll be doing a full RCA of this outage. For those who are interested, on the first Wednesday of each month, I write a blog post on our availability. Most recent one is here…
Thanks for taking the time to personally give a status update while things are on fire. I hope you and all the others who are dealing with this emergency will have an especially restful weekend.
Earlier quoted context omitted.
They should have went to the mat over that ridiculous, pc, woke bullshit decision.
Glancing over your comment history, this seems to be a trend for you; being intentionally abrasive/provocative for no reason. And you get downvoted to oblivion as a result, so I'd hope you understand by now that your comments aren't appreciated. Obviously you're entitled to your opinion and welcome to share it, but don't be surprised when that begins having repercussions on your career, and don't be surprised when pe…
The fact is, tech is now under the influence of the woke mob, and I believe that's objectively bad for tech, the people who work in tech, and the prospect of solving REAL problems with tech.
Hi all - I'm the head of engineering at GitHub. Please accept my sincere apology for this downtime. The cause was a bad deploy (a db migration that changed an index). We were able to revert in about 30 minutes. This is slower than we'd like, and we'll be doing a full RCA of this outage. For those who are interested, on the first Wednesday of each month, I write a blog post on our availability. Most recent one is here…
Earlier quoted context omitted.
Glancing over your comment history, this seems to be a trend for you; being intentionally abrasive/provocative for no reason. And you get downvoted to oblivion as a result, so I'd hope you understand by now that your comments aren't appreciated. Obviously you're entitled to your opinion and welcome to share it, but don't be surprised when that begins having repercussions on your career, and don't be surprised when pe…
Glancing at your comment history, I can see that you believe your opinions are edicts and nobody can express differing opinions without you pulling your quasi moralistic hall monitoring trope. The fact is, tech is now under the influence of the woke mob, and I believe that's objectively bad for tech, the people who work in tech, and the prospect of solving REAL problems with tech.
Hi all - I'm the head of engineering at GitHub. Please accept my sincere apology for this downtime. The cause was a bad deploy (a db migration that changed an index). We were able to revert in about 30 minutes. This is slower than we'd like, and we'll be doing a full RCA of this outage. For those who are interested, on the first Wednesday of each month, I write a blog post on our availability. Most recent one is here…
@keithba I have build a - private - GitHub action around https://github.com/sbdchd/squawk - for Postgres - that lints all our migrations files on each PR. The action extract raw SQL from the codebase and pass them into squawk. It catches many exclusive locks migration or missing `index concurrently` that would otherwise have been release to production and causing downtime or degraded service. Maybe something you shou…
Even so, it's always possible for an engineer to submit a schema change which is detrimental to performance. For example, dropping an important index, or changing it such that some necessary column is no longer present. Linters simply cannot catch some classes of these problems, as they're application/workload-specific. Usually they must be caught in code review, but people make mistakes and could approve a bad change.
Disclosure: I'm the author of Skeema, but have not worked for or with GitHub in any capacity.
[1] https://github.com/github/gh-ost
[2] https://github.blog/2020-02-14-automating-mysql-schema-migra...
Hi all - I'm the head of engineering at GitHub. Please accept my sincere apology for this downtime. The cause was a bad deploy (a db migration that changed an index). We were able to revert in about 30 minutes. This is slower than we'd like, and we'll be doing a full RCA of this outage. For those who are interested, on the first Wednesday of each month, I write a blog post on our availability. Most recent one is here…
I was just griping on Twitter yesterday about how many developers won't immediately revert an update that causes downtime, but will actually spend time trying to solve the problem while Rome burns. Thank you for not doing that.