Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we are not leaving the cloud

about.gitlab.com

Post reply on HN
Print view View original

111–120 of 216 posts

Re: Why we are not leaving the cloud

#111

Post by tw04 »

dragonwriter wrote: post #103
tw04 wrote: post #98

Earlier quoted context omitted.

It isn't, and it's not even close. Unless you're running at less than 40% efficiency the cloud is more expensive for infrastructure running 24/7. If you're a shop with 1 server and no IT guy? Sure, probably makes sense to use the cloud - although I'd argue you'd be better off with managed hosting so you actually have a number to call if something breaks. An organization of even medium size? Not a chance.

> Unless you're running at less than 40% efficiency the cloud is more expensive for infrastructure running 24/7. Sure, paying extra for a dynamically scalable service is inefficient if your resource needs are flat 24/7/365, but is that realistically the case? (OTOH, one must not forget that there is a cost to engineer systems to take advantage of dynamic scaling to minimize costs while meeting requirements.)

In most enterprises yes. They've got a consistent flat workload 90% of the time. I'm willing to bet gitlab is the exact same way other than a tiny portion of the environment. And, as you mentioned, writing an app from scratch to take advantage of instant scaling is extremely difficult, and sometimes impossible depending on what statefulness is required.

Re: Why we are not leaving the cloud

#112

Post by kogepathic »

234dd57d2c8dba wrote: post #17
Dangeranger wrote: post #3

It's interesting that the first half of the explanation is largely quotes from a prior HN post's responses. Does anyone else feel a bit weird seeing off hand comments getting quoted in the explanation for a business decision? I guess we should all get more accustomed to our public input carrying weight in the zeitgeist. Gitlab's "develop in the open" nature really shows through here. I am not saying that's bad, it's…

No surprise here. Honestly, their team seem to be lower skill or less experienced than I'd have thought. I would never approve a production system as expansive as Gitlab's to only have two databases in a cluster. That is asking for trouble, and any {sys,db}admin worth their salt will tell you the same. As soon as you need to do anything on one database, you've just lost your cluster policy. The lack of automation, es…

> I would never approve a production system as expansive as Gitlab's to only have two databases in a cluster. That is asking for trouble, and any {sys,db}admin worth their salt will tell you the same. As soon as you need to do anything on one database, you've just lost your cluster policy.

I do agree with you it reflects poorly on GitLab for only having a primary and replica, with broken backups.

BUT, they are a startup, and they need to be laser focused on growing the company and securing funding for the next quarter so they can keep the lights on.

This kind of pants on fire growth in a startup often (always?) comes at the cost of redundancy and best practices. If you stop to make your platform bulletproof instead of the new features you promised customers/investors, you die.

I'm not saying this is an excuse for them to permanently shrug off making their platform more redundant and engaging in best practices. But, as with many startups, they're focused on delivering features as fast as possible to grow their user base.

I think, and hope, that their recent outage has been the experience they needed to prioritize their shift toward more redundancy and best practices.

Re: Why we are not leaving the cloud

#113

Post by degenerate »

Rapzid wrote: post #104
Dangeranger wrote: post #3

It's interesting that the first half of the explanation is largely quotes from a prior HN post's responses. Does anyone else feel a bit weird seeing off hand comments getting quoted in the explanation for a business decision? I guess we should all get more accustomed to our public input carrying weight in the zeitgeist. Gitlab's "develop in the open" nature really shows through here. I am not saying that's bad, it's…

Well... Here is something very interesting. Myself and others were "cautioning" very heavily against moving to self-hosted gear. It wasn't until their CTO talked to somebody in person he had respect for that they flipped over to staying in the cloud. This to me calls into question the entire idea of crowd-sourcing decisions and plans like this. It was a single person who moved the needle, and now there is retroactive…

Recommending someone to build in the cloud is an easy and safe recommendation to make. Therefore, more people will go that route when asked what you should do.

Pretend for a second you plucked someone from a royal family that never prepared their own food, and they asked you how to make food. Are you going to recommend they purchase land, start a farm, grown their own vegetables, and raise their own cattle? Hell no. You tell them to go to the grocery store (cloud) and if they are desperate, they can purchase some pre-made food at restaurants (SaaS providers).

When you don't know anything about a person's exact needs or abilities, making the safest recommendation is easiest, and therefore it proliferates.

I hate the cloud but I know how to do what I want on my own without it. You don't know that about other people.

Re: Why we are not leaving the cloud

#114

Post by 234dd57d2c8dba »

kogepathic wrote: post #112
234dd57d2c8dba wrote: post #17

Earlier quoted context omitted.

No surprise here. Honestly, their team seem to be lower skill or less experienced than I'd have thought. I would never approve a production system as expansive as Gitlab's to only have two databases in a cluster. That is asking for trouble, and any {sys,db}admin worth their salt will tell you the same. As soon as you need to do anything on one database, you've just lost your cluster policy. The lack of automation, es…

> I would never approve a production system as expansive as Gitlab's to only have two databases in a cluster. That is asking for trouble, and any {sys,db}admin worth their salt will tell you the same. As soon as you need to do anything on one database, you've just lost your cluster policy. I do agree with you it reflects poorly on GitLab for only having a primary and replica, with broken backups. BUT, they are a star…

Right, it's hard to grow the company when you destroy your only DB servers and your backups don't work. They are lucky they follow yet another bad practice of having their customer's data in some random stage environment, so they could restore it from there.

There's no excuse for not writing a bash script or ansible playbook when you're spinning up the most important machines that hold the entirety of your business so you can spin up a 3rd, 4th, or 5th, hot standby. I've done it many times. It takes about a day to write it, a day to find the bugs, and a day to test it out. Take a week your first time just to make sure.

I don't see any valid excuse to not have a 3rd database server. This is not complicated, it's just lazy and/or cheap.

Re: Why we are not leaving the cloud

#116

Post by viraptor »

problems wrote: post #73
mbesto wrote: post #50

Earlier quoted context omitted.

This is incredibly misguided. > The cloud as a money saving venture is and always has been a damn lie. I personally have conducted migration savings assessments for companies going from data centers (owned/lease hardware) to cloud services. I can tell you with 100% certainty that the savings are there and have seen financial proof of the savings. It's either that, or I'm a liar.

Okay, help me out here. Here's my math: I move about 10 TB of bandwidth a month, I currently am in a DC that offers me 30TB on gigabit for $90/mo for my 1U box that hosts my small business and a few development systems - my server cost me a total of $1500 with 72GB of RAM and 4x3TB drives in RAID10, so I have 6TB of disk that's about at 50% capacity. To go to Amazon EC2 let's see... 64 GB of RAM for $0.862 should do…

3 things I think you should consider in this situation.

1. Reserved instances. If you've got a running business and are expecting to be around for 1-3 years, you get 40-60% discount on that 4xlarge.

2. Do you want to move the service exactly as it is? Maybe you don't need the large ebs? Maybe you can rewrite the storage to S3 instead which is much cheaper? Do you have heavy, sporadic tasks that you can move out of your main system and into lambda+queue and use a smaller instance?

3. How much do you spend on people to monitor the hardware, source and replace the disks, do firmware updates, etc. ? How much on external services to monitor your box which could be replaced with integrated AWS solutions at free tier?

Basically what I'm trying to say is: if you just lift your current system and move it to AWS, you're ignoring lots of opportunities. You need to consider much more than a 1:1 hardware requirements migration.

Re: Why we are not leaving the cloud

#117

Post by true_religion »

scott_s wrote: post #60
tbrowbdidnso wrote: post #34

So everyone says the cloud is the future. I get it. But is this the truth, or what all the tech giants want people to believe? Don your foil hats for a moment and listen to me. All the original internet companies run their own hardware. They rent out excess production capacity to us peons in the form of cloud services. These companies that all run their own hardware exclusively are telling everyone that it's stupid t…

The actual streaming video from Netflix does not come from Amazon's systems. From https://arstechnica.com/information-technology/2016/02/netfl... , "Netflix operates its own content delivery network (CDN) called Open Connect. Netflix manages Open Connect from Amazon, but the storage boxes holding videos that stream to your house or mobile device are all located in data centers within Internet service providers' netwo…

> If you wanted to compete with Netflix, you would need to do a lot more than uses Amazon's cloud services.

Yes you'd need multi-million dollar contracts with all of the major studios. Netflix isn't as much a technical problem as it is a business problem.

HBO has a far worse network, UI, and technology in general but its still competitive simply because it has Game of Thrones and other shows.

Re: Why we are not leaving the cloud

#118

Post by problems »

zip1234 wrote: post #107
problems wrote: post #73

Earlier quoted context omitted.

Okay, help me out here. Here's my math: I move about 10 TB of bandwidth a month, I currently am in a DC that offers me 30TB on gigabit for $90/mo for my 1U box that hosts my small business and a few development systems - my server cost me a total of $1500 with 72GB of RAM and 4x3TB drives in RAID10, so I have 6TB of disk that's about at 50% capacity. To go to Amazon EC2 let's see... 64 GB of RAM for $0.862 should do…

What about the personnel costs? Also, what about the instantaneous bandwidth. If you have huge demand spikes where you need a lot of bandwidth at one time, can your system handle it?

"What if we grow 20x overnight" is basically magical thinking, that's about what it'd take to really cause problems, the odds of it occurring are dirt low. Most of the time it'll lead you to waste money. Certainly not enough to account for nearly that price increase. Especially given that you can still rent dedicated or cloud servers temporarily to accommodate in the event that something like that does happen.

As for personnel, if I had to, I'd hire mostly devops people or pay freelancers for jobs as needed until hitting a limit where you have to dedicate a large portion of someone's time to it and then repeat the math - I fully suspect that this would mean sticking with bare metal. I suspect a 2 or 3 person group dedicating their time to managing only hardware and basic infrastructure for something like Kubernetes could probably handle hundreds of servers without issue. Even if that averages out to an additional cost of $1000 per server per year it'd only be approximately a $80 per server month, nothing like the increase of going to EC2. And renting entire racks gets cheaper than the individual colocation I'm paying for at the moment. It'd surely be interesting

Re: Why we are not leaving the cloud

#119

Post by nailk »

user5994461 wrote: post #110

I love all the comments trying to explain that managing your own hardware, storage, network and backup cannot take that much of your time. Not only it does take a huge amount of time, but it takes so much time that it require multiple dedicated highly-skilled people. Last but not least, the last article about Gitlab was about a major outage where they had as much as 6 out of 6 backups unusable! The fact that they sti…

Why do you think that if you don't use a cloud service you have to own hardware? You don't have to do it.

Dedicated server providers will manage your hardware, that's it.

Re: Why we are not leaving the cloud

#120

Post by throwaway2048 »

praneshp wrote: post #54
mbesto wrote: post #50

Earlier quoted context omitted.

This is incredibly misguided. > The cloud as a money saving venture is and always has been a damn lie. I personally have conducted migration savings assessments for companies going from data centers (owned/lease hardware) to cloud services. I can tell you with 100% certainty that the savings are there and have seen financial proof of the savings. It's either that, or I'm a liar.

Hey, you are a liar. He is very sure that the cloud saving money has always been a damn lie! To hell with your data. /s

what data?
Post reply on HN