Live data from Hacker News

Why we are not leaving the cloud

about.gitlab.com

11–20 of 216 posts

Re: Why we are not leaving the cloud

#11

It's interesting that the first half of the explanation is largely quotes from a prior HN post's responses. Does anyone else feel a bit weird seeing off hand comments getting quoted in the explanation for a business decision? I guess we should all get more accustomed to our public input carrying weight in the zeitgeist. Gitlab's "develop in the open" nature really shows through here. I am not saying that's bad, it's…

I don't think they were using this as justification or explanation in their eventual reversal. I think they provided these quotes as interesting tidbits. Kind of like pull-out quotes on an article. You can ignore them entirely and still get the gist of the post.

On a more general note: It's got to be incredibly hard to do what GitLab does with their extreme transparency. I feel like we have to be careful about reading too deeply into things and nitpicking their culture or process. HN is full of "expert" advice, much of it being terrible. They weighed their options, invited feedback, then made a decision.

I appreciate what GitLab does in being so transparent. None of us are owed explanations or insight into how they operate, yet they go out of their way to provide it. Kudos to syste and his team!

Re: Why we are not leaving the cloud

#12

Hmm. Nearly all the commenters seem to be "extreme", i.e. either full-cloud or full-bare-metal. Why not go hybrid? Do your baseline load with bare-metal and your spike load with AWS, GCE or Azure Cloud.

For most use cases that will complicate the system. Keeping everything the same and simple is preferable.

Re: Why we are not leaving the cloud

#13
What about managed dedicated servers? You don't need to go all in full metal, and be responsible for networking and swapping disks and all that.

I know it's more than a handful of deployment recipes, but it's not like they don't manage their cloud instances either...

This article is a false dichotomy with a seemingly rushed decision.

Re: Why we are not leaving the cloud

#14

It's interesting that the first half of the explanation is largely quotes from a prior HN post's responses. Does anyone else feel a bit weird seeing off hand comments getting quoted in the explanation for a business decision? I guess we should all get more accustomed to our public input carrying weight in the zeitgeist. Gitlab's "develop in the open" nature really shows through here. I am not saying that's bad, it's…

This echochamber has a profound and mostly unwarranted impact on people, myself included.

Re: Why we are not leaving the cloud

#15

Hmm. Nearly all the commenters seem to be "extreme", i.e. either full-cloud or full-bare-metal. Why not go hybrid? Do your baseline load with bare-metal and your spike load with AWS, GCE or Azure Cloud.

Cloudbursting either a database server or their git file storage doesn't sound all that viable, and I'd expect that to be the things that matter/are the bottleneck for Gitlab?

Re: Why we are not leaving the cloud

#16
Why was hybrid not considered? With providers like megaport providing really inexpensive direct connect, this is almost a no brainer.

This may seem harsh, but relying on random commenters shows a huge flaw in how you guys went about this.

Physical environments DO work well, but they do require experience to run them. I have used AWS for as long as its been around, but nothing beats physical environments for "known workloads", as long as you run them EXACTLY like you would run a cloud environment. This is why the hybrid approach is such a great thing. Run what you know well in physical and reap great savings, run what you dont know well cloud ( as well as take advantage of the analytics products), and reap the speed ;)

- This means thinking of physical servers are individual units. - This means redundancy at every level. - This means architecting for failure ( servers and switches do gie ). - This means no shared storage for performance critical parts ( shared storage as below the OS level ). - This means objects stores/sharding/etc as your storage layer. - This means real engineering - And this means exactly the same whether you're physical or not.

I've managed environments of 50 vm's and environments of 50000 physical servers. The methodology is always the same.

and yes, this means you can save some monthly cost, and apply it to staff, that can do a lot more than just maintain this infrastructure.

PS: for those that think that showing up to a datacenter is required, you're doing it wrong. Pretty much any datacenter has hot-hands service, and with the right redundancy, hardware replacement is something you can do at a slower pace.

PPS: im sure someone will nitpick some of my points. The reality is, there is real money savings here. For example, Snapchat spends MORE in cloud infrastructure in 2016/2017 per year, than ALL OF GOOGLE did in 2012... think about that for a second... even netflix runs openconnect to push bits..

Re: Why we are not leaving the cloud

#17

It's interesting that the first half of the explanation is largely quotes from a prior HN post's responses. Does anyone else feel a bit weird seeing off hand comments getting quoted in the explanation for a business decision? I guess we should all get more accustomed to our public input carrying weight in the zeitgeist. Gitlab's "develop in the open" nature really shows through here. I am not saying that's bad, it's…

No surprise here. Honestly, their team seem to be lower skill or less experienced than I'd have thought.

I would never approve a production system as expansive as Gitlab's to only have two databases in a cluster. That is asking for trouble, and any {sys,db}admin worth their salt will tell you the same. As soon as you need to do anything on one database, you've just lost your cluster policy.

The lack of automation, especially around validating db backups, failover (not having the failover process scripted and tested is _begging_ to have a nightmare at 2am where you're reading documentation on how to fail over a db), etc.

The simple thing of having your hostname / $PS1 say the machine's purpose could have stopped this. All prod machines have a bright red PS1 and a clear name of ----.corpnet in my setups.

All of this is reflected in their discussion style, meeting style, etc. Ad-hoc, not very carefully designed, random off-hand comments. Obviously a young team with a lot to learn. Nothing wrong with that, but a lot of customers are relying on their skills! Learn quick!

Re: Why we are not leaving the cloud

#19
post #4

A bit disappointed in the article, as the bulk of it is simply quoting various viewpoints, some of which disagree with each other. I'd be much more interested in the thought process that occurred for them to reach the conclusion they did. What internal doubts did they raise? What possible solutions to those doubts did they consider? Things like that.

The reasoning behind their decision is linked near the top of the article, under "Sid and the team decided": https://gitlab.com/gitlab-com/infrastructure/issues/727#note...

Re: Why we are not leaving the cloud

#20
HN is great way to get opinion about a decision. Although sometimes the criticism is way more than the positivism and support.

Post about "We use X and is great" will usually attract lots of "X is bad, use Y" comments.

I really hope that GitLab took a well informed decision based on arguments and counter-arguments.

Post reply on HN