I’m at Google, we have a million + file codebase. Every piece of code is snapshotted (no need to commit, automatic snapshot whenever a change is made). Every line of code has its own unique URL (this is for your branch too, not just overall). Background tests running nonstop hourly. Never seen any downtime. The infra at Google is insane
GitHub was having issues
161–170 of 231 posts
Re: GitHub was having issues
#162Earlier quoted context omitted.
I used GitHub for years in the tech industry, then went into the games industry and used BitBucket and hated it, thought it was such a downgrade. Now I'm back in the tech industry and using GitHub, and I miss BitBucket.
Starting to pretty much desire tools that can do nothing but are just fast at their core competency. Looking at PRs in github and then when toggling to the "files" tab it chocking up or being like "I don't want to display this file because its more than 100 lines" is like wtf you're whole point is to show me modified files.
Come to think of it, this might be great advice for life in general: do one thing very well, and be modular (aka play well with others)
Re: GitHub was having issues
#163Enterprise customers, remember to email your sales rep and ask for them to report on their contracted uptime with you that you are allowed to do as per contract. They wont do this unless you ask hoping you don't notice the outages. It creates lots of internal pain - they have no automation internally for reporting on this. This is the only way anything will ever change. GitHub is _easily_ the most unreliable SaaS pro…
> GitHub is _easily_ the most unreliable SaaS product. Some of us are stuck using Atlassian and BitBucket and it is by far worse in every way.
Re: GitHub was having issues
#164I'm a fan of github but lately i'm seeing a lot of issues like these... Also they don't have yet support for IPv6 (surprising).
Given that they are probably at least partly on Azure, this makes it less surprising because Azure has the worst IPv6 implementation of the 3 large cloud providers.
What is wrong/missing?
Re: GitHub was having issues
#165Enterprise customers, remember to email your sales rep and ask for them to report on their contracted uptime with you that you are allowed to do as per contract. They wont do this unless you ask hoping you don't notice the outages. It creates lots of internal pain - they have no automation internally for reporting on this. This is the only way anything will ever change. GitHub is _easily_ the most unreliable SaaS pro…
Companies should automate this. Write their own outage monitoring, feed the results, plus the cumbersome format you have to send to the provider, into an LLM, have it spit out an email requesting SLA credits or whatever the contract specifies. Probably not worth it for low cost services, but if you’re paying GitHub $x millions per year, maybe it is.
Re: GitHub was having issues
#166Enterprise customers, remember to email your sales rep and ask for them to report on their contracted uptime with you that you are allowed to do as per contract. They wont do this unless you ask hoping you don't notice the outages. It creates lots of internal pain - they have no automation internally for reporting on this. This is the only way anything will ever change. GitHub is _easily_ the most unreliable SaaS pro…
You all should be running a small private git
Most teams could get buy with something like a tiny instance with snapshots every commit
AWS has too much skin in the game to be as unreliable as they used to be.
There's zero reason for a startup to use all these services anymore. The only reason they ever existed was big government manipulation of the labor market through ZIRP
It's far more set n forget to self host git than github will be
Re: GitHub was having issues
#167Earlier quoted context omitted.
What VPS are you using that doesn't come with both IPv4 and IPv6?
Hetzner charges extra for IPv4 address, as I believe most of them do. I know because I went through the same crap.
Re: GitHub was having issues
#168Earlier quoted context omitted.
Companies should automate this. Write their own outage monitoring, feed the results, plus the cumbersome format you have to send to the provider, into an LLM, have it spit out an email requesting SLA credits or whatever the contract specifies. Probably not worth it for low cost services, but if you’re paying GitHub $x millions per year, maybe it is.
Some customers of my product, StatusGator, do this with our API. They can extract the outage data -- including the time when we detect the outage before its acknowledged. And then use that to get SLA credits.
Re: GitHub was having issues
#169Is this a good time to mention forgejo as a self hosted alternative?
But be aware if you intend to host it you will need to protect it from recent AI web scrapers. I use anubis but there are other alternatives.
Re: GitHub was having issues
#170Earlier quoted context omitted.
> GitHub is _easily_ the most unreliable SaaS product. Some of us are stuck using Atlassian and BitBucket and it is by far worse in every way.
Amen, BitBucket outages are regular and showstopping. Self hosting is genuinely more reliable.
I hope I never again have to explain to someone that you can’t just “restore the code from the weekly database backup” because the code is in the file system they just osmosed.