51–60 of 88 posts
What's the best practice for high availability (self-hosted?) repositories? Is there a pass-through proxy for git? Or a leader-follower arrangement that is nice, with a proxy server?
Best practice for me is to hope this happens on Friday afternoon and then take a half day... so all according to plan!
Unfortunately right when we were trying to deploy a hot-fix to production, our CI can't clone the PR to run tests. What do other folks use to avoid this situation? Have a Gitlab instance or similar that you can pull from instead for CI?
Don't use Microsoft?
This is becoming a regular occurrence by now.. I wonder if reliability has become less of a priority. As somebody with little to no experience of running things at scale I’m finding myself attributing this to some form of “move fast and break things”.
good point; I think they might be deploying big features on fridays? as of late I've noticed most of the issues I have seen happen on Fridays.
Unfortunately right when we were trying to deploy a hot-fix to production, our CI can't clone the PR to run tests. What do other folks use to avoid this situation? Have a Gitlab instance or similar that you can pull from instead for CI?
Multiple remotes can help and is certainly something you should have as a backup. However I don't think it solves the root cause which is how the CI is configured.
I'm a firm proponent of keeping your CI as dumb as possible. That's not to say unsophisticated, I mean it should be decoupled as much as possible from the the how of the actions it's taking.
If you have a CI pipeline that consists of Clone, Build, Test, and Deploy stages, then I think your actual CI configuration should look as close as possible to the following pseudocode:
stages:
- clone: git clone $REPO_URL
- build: sh ./scripts/build.sh
- test: sh ./scripts/test.sh
- deploy: sh ./scripts/deploy.sh
This doesn't rule out using CI specific features, such as an approval stage. Just don't mix CI level operations with project level operations.
As a side benefit this helps avoid a bunch of commits that look like "Actually really for real this time fix deployment for srs" by letting you run these stages manually during development instead of pushing something you think works.
More importantly though, it makes it substantially easier to migrate between CI providers, recover from a CI/VCS crash, or onboard someone who's responsible for CI but maybe hasn't used your specific tool.
My heart can't handle another rollercoaster of unicorns for long...
What's the best practice for high availability (self-hosted?) repositories? Is there a pass-through proxy for git? Or a leader-follower arrangement that is nice, with a proxy server?
Believe it or not, we have higher service availability hosting GitLab ourselves than GitHub
I was in the middle of some last minute pre-weekend PR review, and midway I discover it can't actually submit any of my comments. Is there a way to review and save (intermediate) state offline?
I believe Github saves your input to local browser storage, so just make the comments and don't submit them til later.
I wonder if they track Github Status traffic volume as some sort of meta-indicator? Is it even viable? I was futzing around with the description for a PR and hitting save wouldn't update it, yet clicking edit would show the text I expected to see. Suspecting something was up I checked Github Status but it was green across the board. Assuming enough other people hit the same chain of events, could it provide a reliabl…
> I wonder if they track Github Status traffic volume as some sort of meta-indicator? Is it even viable? Sure, the previous decent sized company (~1000+ devs) had that exact metric available. Visits to the status page generally that is. Now whether you could actually correlate that to an increase in errors for a particular component, no so much ;) I'm sure it's totally feasible but it requires a certain amount of dis…
> Now whether you could actually correlate that to an increase in errors for a particular component, no so much ;)
Yep, makes sense. I was picturing a broad "Something Bad Happened, Go Investigate" notification. But I imagine the sensitivity would have to be tuned, especially to account for massive traffic increases from places like HN.
> Even worse, some applications would return a shared error page but internally, I believe it was logged as a 301 redirect until someone spotted it :)
Yikes!
Earlier quoted context omitted.
Best practice for me is to hope this happens on Friday afternoon and then take a half day... so all according to plan!
I just want to be able to run self-hosted CI without it failing due to github
What's the roadblock on doing self-hosted CI without failing due to GitHub? It would be as simple as not using GitHub I think
For those who are interested, on the first Wednesday of each month, I write a blog post on our availability. Most recent one is here: https://github.blog/2021-03-03-github-availability-report-fe...