351–360 of 621 posts
Earlier quoted context omitted.
What if I told you all of society is held together by duct tape? If you're surprised that startups cut corners you're in for a rude awakening. I'm frequently amazed anything works at all.
The real question is what holds together duct tape?
They say that git data (repos and wikis) are not affected... well ... if they would have had their PRs and Issues in git repositories, too... Disclaimer: Worked on a POC for exactly this last semester and going to publish my results in the next few weeks.
https://github.com/docker/datakit/tree/master/ci
Would be simpler if GitHub offered this data in Git format directly.
"The replication procedure is super fragile, prone to error, relies on a handful of random shell scripts, and is badly documented" This is true of many databases, but in my experience is particularly true of postgres. It's a marvelous single-instance product, but I've never really found any replication/ha tech for it that I've been that happy with. I've always been a bit nervous about postgres-backed products for thi…
If we're talking normal replication then I can tell you for certain that the built in replication system is completely rock solid (especially compared to MySQL).
If you're talking multi-master replication, then Citus is pretty solid, but not nearly as solid as replication.
If you're talking statement based replication, well, it's the same as all databases. Here be dragons.
I noticed the issue when I was pushing code earlier today. Hopefully this gets resolved soon. You guys are doing a great job. Keep up the good work!
Thanks, not feeling great about the job we're doing today, but we'll learn from this.
Applying for work with you now, and moving all my stuff to GitLab.
Earlier quoted context omitted.
LOL. GitHub is also a RoR shop.
It might be. I'm not saying it's impossible to scale Rails. It's just very, very hard. Github can do this, because they probably get the best of the best engineers. They even used to have their own, patched Ruby version. Not everyone can afford that.
And as someone working on one of the biggest and oldest Rails codebase out there, I can tell you that in term of scaling, Rails is the least of our concerns.
Sure it's not as efficient, so it's gonna cost you more in CPU and RAM, but it's trivial to scale horizontally. The real worry are the databases, they are fundamentally harder to scale without tradeoffs.
As for the patched Ruby, we used to have one too (but our patches landed upstream so now we run vanilla). It's not about allowing to scale at all. It's simply that once you reach a certain scale, it's profitable to pay a few engineers to improve Ruby's efficiency. If you have 500 app servers, a 1 or 2% performance gain will save enough to pay those engineers salary.
I applaud their forthrightness and hope that it's recoverable so that most of the disaster is averted. To me the most illuminating lesson is that debugging 'weird' issues is enough of a minefield; doing it in production is fraught with even more peril. Perhaps we as users (or developers with our 'user' hat on) expect so much availability as to cause companies to prioritize it so high, but (casually, without really be…
The initial goal of instrumentation should be to provide sufficient cover to a broad area of failure scenarios (database, network, CPU, etc), so that in the event of a failure, you immediately know where to look. Then, once those broad areas are covered, move onto more fine-grained instrumentation, preferably prioritized by failure rates and previous experience. A bug should never be undetectable a second time.
As a contrived example, it was "instrumentation," albeit crudely targeted, that alerted GitLab the problem was with the database. This instrumentation only pointed them to the general area of the problem, but of course that's a necessary first step. Now that they had this problem, they can improve their database-specific instrumentation and catch the error faster next time.
Earlier quoted context omitted.
On a production server this delicate, (eg: production db), I'd replace /bin/rm with a script that moves stuff to /trash. Then, always remember to delete /trash a few days later.
This goes on my machines: sudo apt install trash-cli alias rm="echo This is not the command you want to use" This way I managed to get unlearn my rm trigger-happyness and use trash instead. I had too many incidents already... ':|
Earlier quoted context omitted.
I think it's a staff member. Can't remember first name, Yuri maybe, who is fairly active with the project.
Nope, that would be me.
Earlier quoted context omitted.
On a production server this delicate, (eg: production db), I'd replace /bin/rm with a script that moves stuff to /trash. Then, always remember to delete /trash a few days later.
This goes on my machines: sudo apt install trash-cli alias rm="echo This is not the command you want to use" This way I managed to get unlearn my rm trigger-happyness and use trash instead. I had too many incidents already... ':|
https://bugs.launchpad.net/ubuntu/+source/nautilus/+bug/1442649
The major Azure problems like two years ago were documented by GitLab in similar manner. I find the openess a good thing, even in not so good times. Thumbs up.