Live data from Hacker News

GitHub having issues today

githubstatus.com

71–80 of 89 posts

Re: GitHub having issues today

#71

Earlier quoted context omitted.

Sometimes it's not Ops. Sometimes its crap code. ;) Source: <-- OPs

Clients I worked with: Our service crashed, why? Because you designed and implemented it poorly, that's why. Alternatively: How should I know, you wrote it. If you're ever bored as a developer, switch to operations, you get to be the person developers turn to when they can't code, debug, do logging or security.

Sadly devops will be one of the first to go as AI progresses.

Re: GitHub having issues today

#72

Earlier quoted context omitted.

Sometimes it's not Ops. Sometimes its crap code. ;) Source: <-- OPs

Clients I worked with: Our service crashed, why? Because you designed and implemented it poorly, that's why. Alternatively: How should I know, you wrote it. If you're ever bored as a developer, switch to operations, you get to be the person developers turn to when they can't code, debug, do logging or security.

I've never really been interested in straight development, though I do enjoy the occasional coding session. I'd say at this point, I'm ops through and through, and I feel that the value I add is understanding the systems and what they're doing at a fairly low level. As such, I do sometimes have to help developers out with things I consider surprisingly basic (not usually code exactly), but that's the nature of teamwork I suppose; I'm happy to have a place in things and don't shy away from epithets like "yaml wrangler" or "helpdesk for devs".

Re: GitHub having issues today

#73
post #50

Wouldn't it be wonderful if the most popular version control system was is decentralized? This is achievable, and is the correct solution. This way your git repo could be located on: - GitHub - Your Closet (...) - UCLA's supercomputer - JBOD in Max Planck Institute (...) - GitLab Doing this with a simple file with "[ipfs, github, gitlab]" on it would be revolutionary, especially for data version control, like nn weig…

Can't tell if this comment is sarcastic, but that's exactly what git is: Every clone of the repo is independent, and acts as a full backup. Likewise, a local repo can be pushed to various remotes, there is no inherent strong server-client coupling (even though it's often used in such a way).

Yes but not having to specify every single user who pulls the git repo from GitHub by IP address as an additional remote is a huge win no?

If "ipfs" can be added as a remote, and it automatically pulls from thousands of different devices without having to specify them, that's a pretty big win for redundancy right?

Re: GitHub having issues today

#74

Earlier quoted context omitted.

Git is already decentralized....

So is cryptocurrency, but people still lose their money when exchanges get hacked. :D It's a cry for help. "Make decentralization as convenient as centralization!"

Not sure I understand the 'cry for help's comment. This has absolutely nothing to do with currency here. Associating decentralization with currency is idiotic and completely misses the point.

Well, actually on second thought it does: If ipfs or torrent can be used as a data version control backend, people no longer have to pay $600 to get the very popular and basic dataset of arxiv.org from AWS - under torrent or IPFS it's free. So basically decentralization here means running away from money hungry scams like AWS.

Re: GitHub having issues today

#75

Earlier quoted context omitted.

Clients I worked with: Our service crashed, why? Because you designed and implemented it poorly, that's why. Alternatively: How should I know, you wrote it. If you're ever bored as a developer, switch to operations, you get to be the person developers turn to when they can't code, debug, do logging or security.

Sadly devops will be one of the first to go as AI progresses.

While on the surface I'd agree with you, in reality I think operations people are going to be around longer than developers at this rate.

It's fairly "easy" and relatively safe to let an AI loose on your Java code base and use it to add new features or find bugs. Very few people would let a similar AI roam around production servers and databases.

If you collect enough logs, exceptions/crash dumps, network traffic and so on, you could feed that to the AI and have it tell you why a service crashed. The majority of my job as an operations person is to figure out why something crashed with only a subset of that information and being able to read the code and reason about why current circumstances resulted in the crash or data corruption. Sometimes the job is even to implement the stuff the developers didn't, while not actually touching the code and relying on what the operating system, database, web server or network tells you.

If developers where better, or had more time, more resource then yes, an AI could do the job faster and better. In current environment, operations is pretty safe.

Re: GitHub having issues today

#76

Earlier quoted context omitted.

Sadly devops will be one of the first to go as AI progresses.

While on the surface I'd agree with you, in reality I think operations people are going to be around longer than developers at this rate. It's fairly "easy" and relatively safe to let an AI loose on your Java code base and use it to add new features or find bugs. Very few people would let a similar AI roam around production servers and databases. If you collect enough logs, exceptions/crash dumps, network traffic and…

I sure hope so. But realistically AI will thin the devops herd much more rapidly than "proper" development teams. I already use an LLM to crank out my configs, shell scripts, analyse my logs etc. LLMs are a lot better at these things than full fledged development. I think I am now able to do in days what it would have taken me weeks. My employer does not need to hire as many devops people as it would have pre LLMs.

As you can see you don't need to give AI write access to your production environment.

Re: GitHub having issues today

#77
post #7

The fact that Github has been so unstable for so long is absolutely insane to me. I know ops is hard, but this level of consistent outage points to an endemic problem. Is it the legacy rails/mysql stack that is the largest culprit or is there systemic rot in the engineering org?

More likely, it's efforts to migrate away from the previously solid Rails stack to MS's preferred stack. They've had a long history of this kind of stability issue when migrating or trying to migrate acquisitions from their previous stack to an MS one. This happened with Hotmail (Unix server -> Windows server), LinkedIn (custom cloud -> MS cloud) and others since.

Never heard of linkedin problems before.

Re: GitHub having issues today

#78

Earlier quoted context omitted.

Sadly devops will be one of the first to go as AI progresses.

While on the surface I'd agree with you, in reality I think operations people are going to be around longer than developers at this rate. It's fairly "easy" and relatively safe to let an AI loose on your Java code base and use it to add new features or find bugs. Very few people would let a similar AI roam around production servers and databases. If you collect enough logs, exceptions/crash dumps, network traffic and…

AI can't swap a drive. AI cannot clear a printer jam. AI cannot replace a spinning rust drive in a laptop with an SSD.

Trust me, there will always be OPs people.

Source: 30+ years in Ops/SysAdmin

Re: GitHub having issues today

#79
post #38
post #7

The fact that Github has been so unstable for so long is absolutely insane to me. I know ops is hard, but this level of consistent outage points to an endemic problem. Is it the legacy rails/mysql stack that is the largest culprit or is there systemic rot in the engineering org?

> so unstable for so long Has it? I’ve had hardly any problems. Occasional issues, but rarely have I been impacted to the extent I notice for more than say an hour…. maybe I notice it a couple times a year. My internet access at home is more likely the issue when I hit GitHub issues.

I've experienced a ton of issues, but it's likely because almost every aspect of our operation depends in someway on GitHub -- the repo itself and basic push/pull, or PRs, or webhooks, or actions, or some aspect of status updates, or random API tasks, etc. We use a lot of GitHub.

Re: GitHub having issues today

#80

Earlier quoted context omitted.

Sadly devops will be one of the first to go as AI progresses.

While on the surface I'd agree with you, in reality I think operations people are going to be around longer than developers at this rate. It's fairly "easy" and relatively safe to let an AI loose on your Java code base and use it to add new features or find bugs. Very few people would let a similar AI roam around production servers and databases. If you collect enough logs, exceptions/crash dumps, network traffic and…

DevOps isn't really ops though, right? As in, not product ops. It's ops but for devs, so it's rare for them to have to handle production servers. At least, I hope so. Those would be SRE or even sysadmins, right? I'm not up to date on the usage of the term though haha.
Post reply on HN