211–220 of 236 posts
On the other hand, even if there was some data loss, shouldn't most people have their entire repositories on their drives (at least the ones they're actively working on) so in theory much can be recovered by end-users who are active. The only true worry is inactive users. Not sure if this was discussed much.
Earlier quoted context omitted.
HN can be funny sometimes. GitHub got a lot of hate about a year ago just for not releasing new features. GitLab cost everyone a day because their backup/ops practices were silly, and everyone loves them more. I've screwed up before, and I sympathize/empathize with their ops folks, but this should make us think about plan B in case something like this happens again.
>GitLab cost everyone a day That's just nonsense. In most cases, the local repo should be more than enough to continue work.
I'm not trying to skewer Gitlab, I host an instance at home. I've also nuked 30% of the ports on our openstack cluster and screwed up everyone's day. I admire the transparency. I just wanted to call out HN's reaction, vs way smaller (technical) issues involving GitHub. But someone did point out downthread that HN's not a hivemind, so there is that.
Earlier quoted context omitted.
I think you might be confused. One of the nice things to happen to OSS was the availability of transistors and electricity. Both made it possible to make millions of logic calculations per second without knowing it. :)
I think you might be confused. One of the nice things to happen to OSS was the availability of human beings to actually develop the concept of OSS and any underlying technology.
Earlier quoted context omitted.
Github itself is not open source. The irony.
Yeah, even their enterprise edition code is obfuscated like crazy. For a company that praises open-source so much they really do try to be as closed as possible themselves.
Earlier quoted context omitted.
Well, even if their transparency in failure is admirable, I'm not completely surprised. Just running an apt-get install gitlab gives me around ~350 dependencies. In those dependencies, I see python, ruby, nodejs, redis, postgres. With a little Java, and a little Go, plus some admin scripts written in Perl, the picture would be mostly complete... I may be a little harsh, but when I see a piece of software with so much…
I would be surprised if most startups arent doing something similar behind the scenes.
[Python|Ruby|JavaScript] backend + [JavaScript] frontend + [Postgres|Mongo] database + redis + [AWS|GCP] hosting.
Rarely, I see Java or Go instead for the backend. I cannot recall the last time I saw more than three languages in production for anything nontrivial. I've seen companies significantly larger subsisting on one language for backend (even in microservice architecture!) and one language for frontend. That's not to say there is no sophistication, just that the number of actual technologies in play is slimmer.
This isn't a comment on GitLab's utility or stability, of course. I haven't worked with them in this context, and I'm not a user. I'm just pointing out that, assuming those dependencies are all for GitLab and not e.g. git itself, that is quite a stack to maintain. I don't know if we can extrapolate that to a systemic issue with GitLab that caused a data loss incident, though. That seems uncharitable.
Honestly, I'm completely flabbergasted by this. Five backups, and NONE worked properly? Who made this? The S3 bucket was EMPTY? Has no one ever tested any of these backups? It's not just the impact, which is fairly sizeable in it's own right, but it's the HUGE oversight on their part and the fact they tried to pin part of this on PostreSQL? Credit where it's due; their report/transparency were good if a little unprof…
Untested backup === No backup People like to say they have backups or a "backup procedure", but in my experience almost none of them ever tested the backup... Not even once. 95% of the time "having a backup procedure" just means "we have a replica of some data sitting somewhere with no idea how/if we can restore it, or how long it takes".
Earlier quoted context omitted.
The one who erred (and recovered) once is more valuable than the one who never made mistakes.
There is no basis for that assertion.
Manufacturer A: perfect product. Manufacturer B: omits some pieces, expedites replacements.
People will love and extol the amazing service virtues of B, not A.
Earlier quoted context omitted.
I would counter that they're still in an awful spot because this announcement reeks of incompetence and isn't something you want to hear from the guys you're entrusting with keeping your code safe. It would be like Boeing or Airbus announcing all the safety features on their airliners were non functioning.
The fact that they were upfront and honest about it and that they even live streamed themselves fixing the problems makes me want to use gitlab even more. If anything I have even more confidence in them. You didn't hear a peep from Microsoft when the forced windows 10 upgrade bricked thousands of laptops. Perhaps that's why such a huge portion of developers prefer OSX/Linux to Windows? I've run six businesses over th…
"Test-recover backups" is ops 101. "Monitor your backup process to be sure your backup store isn't empty" is ops 101. "Script your rollouts so you don't have an ops person doing SSH on boxes" is... ok, that one might be ops 102.
This points to a company with almost no understanding of how to operationalize software. There are certain to be far more landmines, possibly even bigger ones. Hiring an ops person to fix these problems is definitely possible - and I sincerely wish GitLab luck getting a competent ops team in place before the next crisis.
Earlier quoted context omitted.
Like I said, they're a remote company, and therefore can hire people in cheaper regions, and don't have to adapt to more expensive markets (like your own). I fail to see how this disproves it, it merely proves that you're not a good choice for them (because you live in a, relatively, expensive region).
Most of their staff is from US/UK, but they just do not want to pay fair wages, which is why I posted the other thread where it was highlighted more.
This argument sounds more selfish than fair to me.