Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we are not leaving the cloud

about.gitlab.com

Post reply on HN
Print view View original

121–130 of 216 posts

Re: Why we are not leaving the cloud

#121

Post by sytse »

kogepathic wrote: post #112
234dd57d2c8dba wrote: post #17

Earlier quoted context omitted.

No surprise here. Honestly, their team seem to be lower skill or less experienced than I'd have thought. I would never approve a production system as expansive as Gitlab's to only have two databases in a cluster. That is asking for trouble, and any {sys,db}admin worth their salt will tell you the same. As soon as you need to do anything on one database, you've just lost your cluster policy. The lack of automation, es…

> I would never approve a production system as expansive as Gitlab's to only have two databases in a cluster. That is asking for trouble, and any {sys,db}admin worth their salt will tell you the same. As soon as you need to do anything on one database, you've just lost your cluster policy. I do agree with you it reflects poorly on GitLab for only having a primary and replica, with broken backups. BUT, they are a star…

For sure it caused us to prioritize more redundancy, for more information please see https://about.gitlab.com/2017/02/10/postmortem-of-database-o...

Re: Why we are not leaving the cloud

#122

Post by drdaeman »

tetha wrote: post #82
neurostimulant wrote: post #71

Earlier quoted context omitted.

Care to share more details? On paper, the cost differences between renting/colocating traditional dedicated servers and using cloud service with similar performance/capacity is huge, so it's pretty unusual to actually save money by migrating to cloud. I'd love to hear where the savings actually come from.

My company has done some research on that, in the webscale bubble. It is wrong to think of the divide as "Cloud" vs "Bare Metal" in a VM for server thing. Disregarding up-front investment, you're comparing rented VMs in the cloud with bare metal boxes, datacenter costs, and most important, manpower. At a small scale, you're paying the cloud less than 2 or 3 competent admins with datacenter and networking experience c…

Cloud may save from hardware/driver issues - by providing an already tested environment, that doesn't (normally) trip on some weirdness in, say, network drivers.

But if the problem is in the software stack, one would still need competent sysadmins/system engineers with skills to diagnose and resolve the issue. When (just a random example) oom-killer wreaks havoc and free(1) insists there's more than half of physical memory still available, it doesn't matter whenever one's in the cloud or not.

I believe, skilled system engineers are still a requirement for any large project, be it in cloud or not.

Re: Why we are not leaving the cloud

#123

Post by cookiecaper »

degenerate wrote: post #113
Rapzid wrote: post #104

Earlier quoted context omitted.

Well... Here is something very interesting. Myself and others were "cautioning" very heavily against moving to self-hosted gear. It wasn't until their CTO talked to somebody in person he had respect for that they flipped over to staying in the cloud. This to me calls into question the entire idea of crowd-sourcing decisions and plans like this. It was a single person who moved the needle, and now there is retroactive…

Recommending someone to build in the cloud is an easy and safe recommendation to make. Therefore, more people will go that route when asked what you should do. Pretend for a second you plucked someone from a royal family that never prepared their own food, and they asked you how to make food. Are you going to recommend they purchase land, start a farm, grown their own vegetables, and raise their own cattle? Hell no.…

I don't think the cloud is the grocery store. It doesn't give you a pre-packaged solution that you can just rip open and immediately access.

Rather, the cloud is like leasing the land from an owner. You still have to till, plant, maintain, and harvest, but you have to pay an annual rent to do so. There are a few things that the landowner is responsible to provide, but most of the day-to-day maintenance responsibility is on the lessee. People generally recognize that while renting is sometimes necessary, it's much nicer to be an owner.

We really need to stop this belief that the cloud is a magic thing that can fix your problems. It splits out the responsibility for actual, real-life hardware maintenance to a hosting company, but other than that, the responsibilities, including the responsibility to take backups, set up security, and design a fault-tolerant application, remain in the cloud user's court.

Cloud is just a way to rent a virtual server; you still have to make the server work correctly.

Re: Why we are not leaving the cloud

#124

Post by cookiecaper »

true_religion wrote: post #117
scott_s wrote: post #60

Earlier quoted context omitted.

The actual streaming video from Netflix does not come from Amazon's systems. From https://arstechnica.com/information-technology/2016/02/netfl... , "Netflix operates its own content delivery network (CDN) called Open Connect. Netflix manages Open Connect from Amazon, but the storage boxes holding videos that stream to your house or mobile device are all located in data centers within Internet service providers' netwo…

> If you wanted to compete with Netflix, you would need to do a lot more than uses Amazon's cloud services. Yes you'd need multi-million dollar contracts with all of the major studios. Netflix isn't as much a technical problem as it is a business problem. HBO has a far worse network, UI, and technology in general but its still competitive simply because it has Game of Thrones and other shows.

Scratch business problem, replace with "legal problem". Copyright law makes it virtually impossible to compete with Netflix. Netflix was only allowed to start because they wedged their way in as a rent-by-mail service.

After pioneering streaming and realizing that the studios were asking themselves why they don't just load up their own computers with movies and get paid to stream them out, Netflix was forced to start its own content division to keep subscribers onboard and prevent the studios from poaching all of their customers by failing to renew contracts for content.

We have a similar problem in online services. It's impossible to compete with Facebook because the law allows them to hold data hostage.

We badly need copyright reform. The current incarnation is impractical in the digital age.

Re: Why we are not leaving the cloud

#125

Post by user5994461 »

dsr_ wrote: post #67
morgante wrote: post #62

Earlier quoted context omitted.

Yes, cloud bandwidth is expensive. I don't disagree with that. But it's not increasing in price. Obviously if you're growing your costs will grow, but that's basically a tautology. Bare metal costs also rise as you grow.

Will you stop calling me a liar if I correct that from "the costs" to "your costs"? I am assuming that you are growing, and I think there's a provable assumption that for high levels of service, hiring someone else to do it is more expensive than in-house.

> hiring someone else to do it is more expensive than in-house.

Except you're gonna need between 10 and 100 people to recreate the service in-house, not just one.

Trying to re-do something from scratch in-house is always more expensive than just buying the product.

Re: Why we are not leaving the cloud

#126

Post by cookiecaper »

bdamm wrote: post #44
tbrowbdidnso wrote: post #39

Earlier quoted context omitted.

There is nothing hard about running hardware. The current VM trend gives you bare metal access. the only difference is you have to plug in your machines, and you get to look at them every once in a while. Really, what software dev hasn't put together a desktop PC and plugged in some Ethernet jacks? That's all you need to Colocate. Your response echos what cloud providers want everyone to think. Hardware is too hard f…

As a professional that came from software engineering let me tell you... that plugging in one machine is easy. Plugging in 150 machines is significantly more challenging. Providing disaster response, backups, stable power, and VLANs to secure services, quickly ratchets up the cost. And nobody wants to pay one engineer per machine to babysit one machine, x 150. That wouldn't even be a good idea, because engineers get…

First, most of the time it's not a question of paying someone to run an entire data center, but rather some racks in someone else's data center. The data center has data center staff who handle a lot of this. It's unfair to pretend that going bare metal entails starting with an empty plot of land and building/managing a whole data center. While that may be the case for the largest companies, even moderately-sized companies won't do that.

When you're a business, you have to pay people to handle hardware no matter what. The question is not whether you need to pay someone to administer the hardware -- the question is who you are going to pay. It can be an employee, contractor, or third-party service company like AWS or remote hands at a colo.

I've seen multiple companies move to cloud merely because they were annoyed at having to go down to the rack to deal with hardware, often making oblique justifications about difficulty when they really meant that they don't like driving down to the colocation center.

Those companies have spent millions more paying AWS than they would've if they just hired a couple of hardware jockeys.

The argument basically becomes a question of whether you know how to hire someone who knows how to deal with hardware. If you don't, it's better to go with Amazon even though they're going to rake you over the coals cost-wise. If you do, then practically speaking, the differences should be small. Your hardware people will automatically fix the hardware issues just like Amazon's hardware people.

The bulk of the work of running servers is at the OS admin level, which is still the customer's responsibility with cloud servers. These arguments about difficulty of administration would work for something like Heroku, but they don't work for Amazon.

Re: Why we are not leaving the cloud

#127

Post by gpapilion »

morgante wrote: post #105
problems wrote: post #77

Earlier quoted context omitted.

Yeah, but I'm just trying to ballpark it. I figured the larger server would be a costsavings over many smaller ones. Most of it does need to remain online at all times including several large databases. Just running my logstash server alone eats 8 GB of RAM and 200+ GB of disk. In bandwidth and storage alone I'm past the cost of the server - so even if I could split this up and keep it mostly offline it still wouldn'…

> I figured the larger server would be a costsavings over many smaller ones. Nope, you pay more for a single large machine than multiple small ones. If your service is truly so stable that you only need to spend 10 hours a year maintaining it (and demand isn't growing), then maybe the cloud isn't right for you. But that's not true for most companies.

Actually pricing at both aws and gcp is linear.

Re: Why we are not leaving the cloud

#128

Post by user5994461 »

drdaeman wrote: post #122
tetha wrote: post #82

Earlier quoted context omitted.

My company has done some research on that, in the webscale bubble. It is wrong to think of the divide as "Cloud" vs "Bare Metal" in a VM for server thing. Disregarding up-front investment, you're comparing rented VMs in the cloud with bare metal boxes, datacenter costs, and most important, manpower. At a small scale, you're paying the cloud less than 2 or 3 competent admins with datacenter and networking experience c…

Cloud may save from hardware/driver issues - by providing an already tested environment, that doesn't (normally) trip on some weirdness in, say, network drivers. But if the problem is in the software stack, one would still need competent sysadmins/system engineers with skills to diagnose and resolve the issue. When (just a random example) oom-killer wreaks havoc and free(1) insists there's more than half of physical…

> I believe, skilled system engineers are still a requirement for any large project, be it in cloud or not.

And that's why cloud will win every single time, forever, on all metrics.

Because the cloud requires less engineers to achieve the same work, as it takes care of the low level hardware work.

Re: Why we are not leaving the cloud

#129

Post by cookiecaper »

morgante wrote: post #69
tbrowbdidnso wrote: post #63

Earlier quoted context omitted.

I know it's a controversial opinion. And probably not even true in a lot of cases... but I'm glad it sounds like I'm out in left field. The echo chamber needs a devil's advocate and sometimes the resulting discussion gets interesting enough that I question my original beliefs. In this case I believe owning my hardware is cheaper for me because I already do. I might be a rarity though, or just old fashioned and wrong

Your opinion isn't really that original. There are plenty of old-school sysadmins who will insist that the cloud is a hoax. They'll fight tooth and nail to avoid it. Eventually, someone fires them and moves to the cloud with significant savings. There are some scenarios where it makes sense to run bare metal, but they're few and far between. If you've evaluated the (capital, operational, and human) costs of cloud vs.…

I've seen businesses move to the cloud and cost themselves millions of dollars in doing so. I'm not sure where you're coming from believing that cloud always offers a massive savings. It's possible that in some circumstances, cloud will be cheaper, but I actually think that more often than not, it's more expensive.

It's ludicrous that you suggest that anyone who is not willing to dive head-first into 100% cloud everything should be fired. The cloud has a place, but there's no reason that cloud should be the only option, as you appear to suggest.

Re: Why we are not leaving the cloud

#130

Post by mahyarm »

In my observing-from-a-distance experience, going bare metal makes sense once you are a certain size and have predictable baseline load to maintain. Many companies that have their own data centers still use cloud services to auto scale and deal with spikes.
Post reply on HN