Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we are not leaving the cloud

about.gitlab.com

Post reply on HN
Print view View original

171–180 of 216 posts

Re: Why we are not leaving the cloud

#171

Post by cookiecaper »

morgante wrote: post #167
cookiecaper wrote: post #136

Earlier quoted context omitted.

Yeah, it's not a direct quote. However, that is how I interpret the sentiment that those who fight the cloud will be fired. You suggest that's an unfair interpretation? You can say "cloud may not be the only option" all you want, but "those who oppose cloud will be fired" ensures that somehow, everyone in your organization will believe the cloud to be a supremely good option. :)

Because I do fully believe that the cloud is the superior solution for most businesses, especially those employing sysadmins. I've seen way too many companies with full-time sysadmins maintaining only a few dozen servers. The significant cost savings come from eliminating those positions.

Who administers the servers after you eliminate those positions? Sysadmins mostly do operating system administration, not hardware maintenance. AWS boots you into an empty operating system that still needs to be configured, secured, backed up, etc. You still need sysadmins! In fact, cloud tends to require more of them because people spawn a lot of instances.

If you have a bunch of idle sysadmins because you thought you'd have to replace a system's drive every week and hired 4 extra hardware-only sysadmins, you can probably fire them without moving to the cloud and save even more money, right?

Re: Why we are not leaving the cloud

#172

Post by morgante »

cookiecaper wrote: post #171
morgante wrote: post #167

Earlier quoted context omitted.

Because I do fully believe that the cloud is the superior solution for most businesses, especially those employing sysadmins. I've seen way too many companies with full-time sysadmins maintaining only a few dozen servers. The significant cost savings come from eliminating those positions.

Who administers the servers after you eliminate those positions? Sysadmins mostly do operating system administration, not hardware maintenance. AWS boots you into an empty operating system that still needs to be configured, secured, backed up, etc. You still need sysadmins! In fact, cloud tends to require more of them because people spawn a lot of instances. If you have a bunch of idle sysadmins because you thought y…

If you can treat your instances as ephemeral, it's a lot easier to manage them.

Re: Why we are not leaving the cloud

#173

Post by cstejerean »

cookiecaper wrote: post #165
philliphaydon wrote: post #162

Earlier quoted context omitted.

I don't understand what this has to do with copyright laws. This has to do with content licensing and studios not wanting to license content...

The copyright laws are what give studios the rights to license that content. Right now, most things published commercially after 1923 are copyrighted and require permission from their rightsholder to distribute. That's practically the entire history of cinema locked up tight. Copyright should be adjusted so that people can make money off of their creations without allowing the culture to be held hostage to corporatio…

It wouldn't matter to Netflix. The really old titles are cheap and Netflix always carries them. The problem Netflix had/had is access to latest releases. When it comes to movies this the bulk of what people want to watch. Since they couldn't guarantee access to new content they had to pay to produce their own new content.

Re: Why we are not leaving the cloud

#174

Post by cstejerean »

qaq wrote: post #150
zip1234 wrote: post #106

Earlier quoted context omitted.

Also can be very expensive. If another IT devops type costs 100k per year. How much will that buy in cloud services and time saved?

150 TB/month of data transfer on AWS will cost you 15k/ month the same thing on dedicated provider will be 500/month. And so on.

That's about the fully loaded cost of one engineer, and you may easily need multiple additional people to run your own hardware instead of using AWS.

Re: Why we are not leaving the cloud

#175

Post by qaq »

cstejerean wrote: post #174
qaq wrote: post #150

Earlier quoted context omitted.

150 TB/month of data transfer on AWS will cost you 15k/ month the same thing on dedicated provider will be 500/month. And so on.

That's about the fully loaded cost of one engineer, and you may easily need multiple additional people to run your own hardware instead of using AWS.

That's fairly modest traffic and does not include cost difference for the compute and persistence layers. In case of GitLab their dev and ops office is in Ukraine so that difference on traffic cost alone would pay for 3 good ops people.

Re: Why we are not leaving the cloud

#176

Post by jamesmiller5 »

sytse wrote: post #45
Rezo wrote: post #30

Earlier quoted context omitted.

I wonder if Amazon's new managed EFS service would make sense? It's exposed as a NFS mount to the OS. The claims are: - Up to thousands of Amazon EC2 instances, from multiple AZs, can connect concurrently to a file system. - Data is stored redundantly across multiple AZs. - Low, consistent latency. - Multiple GBs per second.

EFS is pretty great. But for our use case there are a few drawbacks: 1. It doesn't scale to the size we need for all repositories, so we still need sharding. 2. It is expensive, $300/TB-month 3. You're still sending all traffic over the network, we prefer the latency of local storage we can achieve by developing Gitaly https://gitlab.com/gitlab-org/gitaly

Why develop Gitaly instead of using consensus (etcd, zookeeper, etc.) for writes to fast storage like ssds and the normal git binary? As far as I can tell it's adding a lot of complexity by caching high level RPC calls but still doesn't really address multi-master read after write consistency or coordination.

Re: Why we are not leaving the cloud

#177

Post by extrovert »

dasil003 wrote: post #37
keepper wrote: post #16

Why was hybrid not considered? With providers like megaport providing really inexpensive direct connect, this is almost a no brainer. This may seem harsh, but relying on random commenters shows a huge flaw in how you guys went about this. Physical environments DO work well, but they do require experience to run them. I have used AWS for as long as its been around, but nothing beats physical environments for "known wo…

> This may seem harsh, but relying on random commenters shows a huge flaw in how you guys went about this. Your other points and experience are well taken, but this is definitely too harsh. Think about the nature of GitLab as an open company building dev tools and paying below-market salaries. They have a huge amount of developer mindshare relative to their internal head count. They are not very competitive for hirin…

> They are not very competitive for hiring people with your particular skillset.

There is plenty of ops talent willing to work at a high-profile startup. I have experience building and running PB-scale Ceph clusters, private and public clouds, HA infrastructure covering everything from DB failover to BGP, and interviewed at Gitlab when they announced going on-prem. I asked for half the market rate (because I was more interested in solving the scaling problem than money), but did not get it.

The HR person at the first (non-technical!) interview asked a bunch of programming related questions from a script which caught me completely off guard. One question went along the lines of "tell me about a difficult problem that you had a particularily elegant solution for". I think they've learned by now that ops problems usually can't be solved by a simple algorithm.

In retrospect I'm glad I didn't get it, but they seem to be having issues with more than infrastructure.

As a side note, if there are other startups wanting to move to own hardware but struggling to find the right people, I would love to know where to find you. Non-AWS ops jobs are far and few between these days.

Re: Why we are not leaving the cloud

#178

Post by tbrowbdidnso »

polishTar wrote: post #59
dsr_ wrote: post #40

Earlier quoted context omitted.

The cloud isn't a money-saving tactic. It enables you to test demand for a service for a very low absolute cost. Spin up a tiny machine for $60/year? And you can pay for it by the month, or the hour? That's great. You can find out if anybody is going to buy your thing at all. You can probably get to ramen-profitable. As soon as you can confidently predict sufficient demand, the economically rational decision is to hi…

I'd also argue that "sufficient demand" is a threshold that's been rising very fast over the last couple of years and likely to continue doing so. This also explains why older tech companies disproportionately use their own hardware, even those that aren't all that big. They all made that decision at a time when it made sense and the switching cost is sufficiently high to prevent them from turning back now. I doubt t…

You should remember that the early cloud unicorns are getting super low rates for their name clout as well. Their discounts are passed as extra costs to everyone else.

So while it may be worthwhile if you're huge enough to get a big discount, that doesn't mean it applies to small or medium size companies

Re: Why we are not leaving the cloud

#179

Post by ReidZB »

Dangeranger wrote: post #3

It's interesting that the first half of the explanation is largely quotes from a prior HN post's responses. Does anyone else feel a bit weird seeing off hand comments getting quoted in the explanation for a business decision? I guess we should all get more accustomed to our public input carrying weight in the zeitgeist. Gitlab's "develop in the open" nature really shows through here. I am not saying that's bad, it's…

When I evaluate any tech project for use in production, I always give HN a cursory search for materials. It's not always fruitful, but in the cases where I stumble across something big in the HN space, I pick up valuable intel here...

I still try to research elsewhere too, just to cover my bases, but HN is quite helpful as an archive of knowledge. Especially knowledge about running something in production.

Re: Why we are not leaving the cloud

#180

Post by kogepathic »

lightedman wrote: post #160
kogepathic wrote: post #112

Earlier quoted context omitted.

> I would never approve a production system as expansive as Gitlab's to only have two databases in a cluster. That is asking for trouble, and any {sys,db}admin worth their salt will tell you the same. As soon as you need to do anything on one database, you've just lost your cluster policy. I do agree with you it reflects poorly on GitLab for only having a primary and replica, with broken backups. BUT, they are a star…

"BUT, they are a startup, and they need to be laser focused on growing the company and securing funding for the next quarter so they can keep the lights on." Having run and sold a few startups and turnkey operations, let me tell you - if you don't focus on your core product first and foremost and demonstrate the utmost competency in it, you're just pissing money away and are likely to fail.

> if you don't focus on your core product first and foremost and demonstrate the utmost competency in it, you're just pissing money away and are likely to fail.

I don't have direct experience running/selling a startup, but I have worked for several as an employee.

In my experience, the product is important, but not necessarily the deciding factor in whether the company succeeds. Most of the success is down to the management team (e.g. CEO, CTO, CFO) being able to sell the company (and product) to investors during funding rounds.

Doesn't matter how good the product is, if management can't pitch it to get funding you're dead.

Obviously it is important to have a good product, but just look at Uber for example. Losing billions of dollars per year, with no clear path to profitability, and yet they're valued at something like $60B. Great sales/marketing by their management to investors!

Post reply on HN