Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Have you ever switched cloud?

news.ycombinator.com

Post reply on HN
Print view View original

241–250 of 268 posts

Re: Ask HN: Have you ever switched cloud?

#241

Post by safeerm »

drusepth wrote: post #216

I switched from Heroku to AWS, then eventually back to Heroku. Heroku to AWS was for cost reasons (cut monthly costs by roughly 35%) but wasn't enough savings to justify hiring a devops person. As soon as there were too many issues I didn't know how to fix (setting up everything Heroku offers was hard and likely done wrong, which made ongoing maintenance some level of hell), I switched back to Heroku where the lack o…

This is very, very true. One of our customers described it really well “The nightmare of DevOps kept us from managing AWS directly, but with TinyStacks we can scale a billion+ requests a day in audio advertising leveraging the full power of AWS.”

Re: Ask HN: Have you ever switched cloud?

#242

Post by holografix »

lolinder wrote: post #66
eric4smith wrote: post #27

Moved from Google Cloud -> Digital Ocean -> OVH. Running our own stuff on high powered servers is very easy and less trouble than you think. Sorted out the deploy with a "git push" and build container(s) meant we could just "Set it and forget it". We have a bit under a terabyte of Postgresql data. Any cloud is prohibitively expensive. I think some people think that the cloud is as good a sliced bread. It does not rea…

> Really, I can't see a compelling reason to be in the cloud for the majority of mid-level workloads like ours. I work on a very small team. We have a few developers who double as ops. None of us are or want to be sysadmins. For our case, Amazon's ECS is a massive time and money saver. I spent a week or two a few years ago getting all of our services containerized. Since then we have not had a single full production…

To the peeps running ECS? Why not just straight up AKS or GKE? Have you compared ECS to Cloud Run on GCP?

Re: Ask HN: Have you ever switched cloud?

#243

Post by vidarh »

tinco wrote: post #215
vidarh wrote: post #40

Yes. I once did zero downtime migration first from AWS to Google, then from Google to Hetzner for a client. Mostly for cost reasons: they had a lot of free credits, and moved to Hetzner when they ran out. Their savings from using the credits were at least 20x what the migrations cost. We did the migration by having reverse proxies in each environment that could proxy to backends each place, set up a VPN between them,…

At my previous job we ran a bare metal cluster at Hetzner, and monitoring the hardware was quite an intensive task. Always monitoring hard drives network bottlenecks, CPU usage, etc. This was before K8s, so it might not be comparable to today. Would you say bare metal cost a lot of extra monitoring/maintenance, or is this something you did on the cloud hardware as well anyway? Do you run virtualization on the Hetzner…

I would say cloud costs a lot of extra maintenance. When I did contracting, those of my clients who insisted on AWS tended to be far more profitable clients for me, because they needed help with so much more stuff.

In terms of monitoring, it boils down to picking a solution and building the appropriate monitoring agent into your deployment.

I've run basically anything I run in some virtualized env. or other since ~2006 at least, be it OpenVz (ages ago), KVM, or docker. And that goes for Hetzner too. It makes it easy to ensure that the app environment is identical no matter where you move things. I managed on environment where we had stuff running on prem and in several colo's, on dedicated servers at Hetzner, and in VMs, and even on the VMs we still containerised everything - deployment of new containers was identical no matter where you deployed. All of the environment specific details were hidden in the initial host setup.

Re: Ask HN: Have you ever switched cloud?

#244

Post by vidarh »

teh_klev wrote: post #218
vidarh wrote: post #40

Yes. I once did zero downtime migration first from AWS to Google, then from Google to Hetzner for a client. Mostly for cost reasons: they had a lot of free credits, and moved to Hetzner when they ran out. Their savings from using the credits were at least 20x what the migrations cost. We did the migration by having reverse proxies in each environment that could proxy to backends each place, set up a VPN between them,…

I hate to be the bean counter, but what was the true cost ultimately? As you and your team's cost as resources. It's nice that you ended up with a provider agnostic capability to deploy anywhere, but none of that was free in terms of ownership costs to get there.

I was the only person doing the migration work and setting up the HA setup and documenting it for them to take over. My fee for setting it up and doing it accounted for that 20x difference. Hetzner in the end was far cheaper for them to run including the devops work they contracted to me. I effectively got paid to reduce my future earnings from them. But that was fine; when I was doing contracting, that was a big part of my pitch - that I'd help drive down their costs and pay for myself many times over - if people were in doubt I'd offer to take payment as a percent of what they saved.

So, no, it wasn't free, but it saved them far more money than it cost them, both the initial transition and in ongoing total cost of operation.

In fact, my first project for them was to do extensive cost-modelling of their development and operations.

Re: Ask HN: Have you ever switched cloud?

#245

Post by vidarh »

justsomehnguy wrote: post #204
vidarh wrote: post #126

Earlier quoted context omitted.

Unfortunately not, but it's surprisingly straight-forward, apart from the database bit, but here's a bit more detail from memory. There are many ways of doing this and some will depend strongly on which tools you're comfortable with (e.g. nginx vs. haproxy vs. some other reverse proxy is largely down to which one you know best and/or already have in the mix) [Today I might have considered K8s, but this was before tha…

> If you want true zero-downtime migration there's a challenge It is astounding how many people require 24/7 ops... while working 8/5. Otherwise this comment is an exemplar on how things should be done. My take on this is what OP is a sysadmin, not a dev. *smug smile*

> It is astounding how many people require 24/7 ops... while working 8/5.

In this case the client had an actually global audience. They could have afforded downtime for the actual transition, but it was a usual test for the high availability features that mattered for them.

I do agree with the overall principle, though - a whole lot of people think they need 24/7 and can't afford downtime, yet almost all of them are a lot less important than e.g. my bank, which do not hesitate to shut down their online banking for maintenance now and again. As it turns out, most people can afford downtime as long as it's planned and announced. Convincing management of that is a whole other issue.

> My take on this is what OP is a sysadmin, not a dev. smug smile

Hah. I'd say I was devops before devops was a thing. I started out writing code, but my first startup was an ISP where I was thrown head-first into learning networks (we couldn't afford to pay to have our upstream provider help set up our connection, so I learnt to configure cisco routers while having our provider on the phone and feigning troubleshooting with a lot of "so what do you have on your side?") and sysadmin stuff, and I've oscillated back and forth between operations and development ever since. Way too few developers have experienced the sysadmin side, and it's costing a lot of companies a lot of money to have devs that are increasingly oblivious to hardware and networks.

Re: Ask HN: Have you ever switched cloud?

#246

Post by vidarh »

YorickPeterse wrote: post #117

At GitLab we went from AWS to Azure, then to Google Cloud (this was a few years ago). AWS was what we started with, and I think like most companies very little attention was paid to the costs, setup, etc. The result was that we were basically setting money on fire. I think at some point Azure announced $X in free credits for YC members, and GitLab determined this would save us something like a year's worth in bills (…

I love Hetzner, and host most of my own stuff on it. And it takes so little to be prepared to move. E.g. a basic service discovery mechanism, a reverse proxy and putting things in containers and you can migrate anywhere. Now that Hetzner has some cloud features too I see even less reason to go elsewhere (though in my current job we use AWS, but we use AWS with the explicit understanding that we're low volume - currently mostly running internal tools - and can afford the premium; if we needed to scale I'd push for putting our base load somewhere cheaper, like Hetzner)

One additional suggestion to people considering bare metal: Consider baking in a VPN setup from the start, and pick a service discovery mechanism (such as e.g Consul) that is reasonably easy to operate across data centres. Now you have what you need to do migration if you need to, but you also have the backbone to turn your setup into a hybrid setup that can extend into whichever cloud provider you want too.

A reason for wanting that is that one of the best ways I've found of cutting the cost of using bare metal even further is to have the ability to handle bursts by spinning up cloud instances in a pinch. It allows you to safely increase the utilisation levels of your bare metal setup substantially with according cost savings even if you in practice rarely end up needing the burst capability. It doesn't even need to be fully automated, as long as your infra setup is flexible enough to accommodate it reasonably rapidly. E.g. just having an AMI ready to go with whatever you need to have it connect to a VPN endpoint and hook into your service discovery/orchestration on startup can be enough.

Re: Ask HN: Have you ever switched cloud?

#247

Post by vidarh »

ChasingEchoes wrote: post #97
exfascist wrote: post #80

Earlier quoted context omitted.

In fact, if you're using Linux on your workstation you'll use the same skills locally as you do on the VPS/bare metal (depending on your scale.) Arguably "cloud" services need more sysadmin skills, not less.

Thats a very big if. I have yet to work with a $corp that uses Linux for workstations. Overwhelming majority uses Windows. Some use macOS. The ocasional developer that uses linux will usually be in a VM, or if IT policies allow, WSL. So yeah, running cloud services doesnt require sysadmin skills, unless you assume copy pasting from oficial documention "sysadmin skills".

In 27 years, I've had exactly two jobs where I didn't have Linux on my desktop, for a total of 5 out of those 27 years. In both cases, I still did all of my dev work on Linux.

It boils down to what kind of jobs you look for.

> So yeah, running cloud services doesnt require sysadmin skills, unless you assume copy pasting from oficial documention "sysadmin skills".

If that's the extent of how you're managing your cloud setup, then I could equally argue running bare metal servers doesn't require sysadmin skills either. When I did contracting, a large part of my income was to come in and clean up after people had relied on "copy pasting from official documentation" as a substitute for actual ops.

Re: Ask HN: Have you ever switched cloud?

#248

Post by vidarh »

lolinder wrote: post #205
kissgyorgy wrote: post #102

Earlier quoted context omitted.

You still need to do backups, a database backup is just one part of that, if you are not following the 3-2-1 rules and don't test your restore mechanism, you don't have reliable backups. Those things you listed are sill sysadmin tasks in my eyes, and you are doing them, validating my point. You still have to track storage space, either because you are paying for it and need to expand when necessary, or you have to ma…

I tell you that ECS has eliminated a ton of extra work for my team for a bargain price, and your response is "but you still have to do x, y, and z!" It's like saying that I shouldn't buy a dishwasher because I'd still have to wash the pots. Yes, we still need to do some sysadmin-y tasks. But ECS handles so many of them that we actually have the time, energy, and knowledge to take care of the few that remain. (As an a…

> As an aside, keeping language and OS versions up to date becomes a development task rather than an ops task when running Docker + ECS.

It's a development task with a proper bare metal setup too.

Re: Ask HN: Have you ever switched cloud?

#249

Post by teh_klev »

vidarh wrote: post #244
teh_klev wrote: post #218

Earlier quoted context omitted.

I hate to be the bean counter, but what was the true cost ultimately? As you and your team's cost as resources. It's nice that you ended up with a provider agnostic capability to deploy anywhere, but none of that was free in terms of ownership costs to get there.

I was the only person doing the migration work and setting up the HA setup and documenting it for them to take over. My fee for setting it up and doing it accounted for that 20x difference. Hetzner in the end was far cheaper for them to run including the devops work they contracted to me. I effectively got paid to reduce my future earnings from them. But that was fine; when I was doing contracting, that was a big par…

Ah, now I understand. Sorry it hadn't joined up in my brain you were providing this as a service as a third party. Thanks for the explanation.

Re: Ask HN: Have you ever switched cloud?

#250

Post by dx034 »

philliphaydon wrote: post #82
dx034 wrote: post #42

Earlier quoted context omitted.

I'm still not a fan of the second way. If you develop all your software to tightly integrate with AWS, you might save time developing the software but create a huge amount of technical debt. Managing your own infrastructure (with dedicated servers, so no hardware management) isn't too hard, even if you're a small shop. And managing a fleet of AWS services isn't necessarily less work. Maybe there's a reason all ads fo…

It’s not technical debt if it’s making you money. I would much rather solve the business problems than managing services myself.

It's technical debt if you may need to change your product because a third party makes changes. The beauty of software with little dependencies is that you can run decades old software on a system just fine with no need to regularly refactor.

I know how tedious it is to maintain decades old enterprise Java software, but from a cost perspective, it makes much more sense to keep those rather than constantly refactor to chase the newest trend.

As an example, if you had a software that was written 20 years ago to store data in a relational DB, updating it to work with current versions of that database system won't be much work (if any). If you rely on managed services, I wouldn't be too sure that you get away that easy.

Post reply on HN