Live data from Hacker News

Ask HN: Have you ever switched cloud?

news.ycombinator.com

191–200 of 268 posts

Re: Ask HN: Have you ever switched cloud?

#191
post #126

Earlier quoted context omitted.

Would you have a write up in more detail of what you did, even high level. Seems cool thing to do

Unfortunately not, but it's surprisingly straight-forward, apart from the database bit, but here's a bit more detail from memory. There are many ways of doing this and some will depend strongly on which tools you're comfortable with (e.g. nginx vs. haproxy vs. some other reverse proxy is largely down to which one you know best and/or already have in the mix) [Today I might have considered K8s, but this was before tha…

Really well done keeping this simple!

It's also another one of those situations where good design principles and coding practices pay off. If the app is a tangled mess of interconnected services, scripts, and cron jobs this kind of transition won't be possible.

Re: Ask HN: Have you ever switched cloud?

#192

Earlier quoted context omitted.

Interesting that you trust Google more in this regard. Given Google's terrible history of deprecating products, I would not trust any "real world" business to any of their services. Also, at some point I was playing with Serverless in both Google and AWS. Google's Serverless examples were broken (Google cloud was returning 500 errors) while the same stuff in AWS worked smoothly. That left me with a bad taste.

>Dozen of decade+ billon+ customer contracts at this point make shutting GCP down a silly thought. Really crappy that the demos sucked tho, sorry :(

https://www.aljazeera.com/economy/2020/7/8/google-shut-down-...

https://cloud.google.com/support/docs/shutdown

Also, there have been plenty of Google products with paying customers that Google has shot down.

At this point, I have 0 credibility in Google.

Re: Ask HN: Have you ever switched cloud?

#193
post #40

Yes. I once did zero downtime migration first from AWS to Google, then from Google to Hetzner for a client. Mostly for cost reasons: they had a lot of free credits, and moved to Hetzner when they ran out. Their savings from using the credits were at least 20x what the migrations cost. We did the migration by having reverse proxies in each environment that could proxy to backends each place, set up a VPN between them,…

Several years ago I had a string of clients who also came into enough Google Cloud credits to make a switch worthwhile.

For these companies it wasn't a problem to have a few minutes of downtime, so the task was simply recreating their (usually AWS) production environment in Google Cloud.

Re: Ask HN: Have you ever switched cloud?

#194
post #40

Yes. I once did zero downtime migration first from AWS to Google, then from Google to Hetzner for a client. Mostly for cost reasons: they had a lot of free credits, and moved to Hetzner when they ran out. Their savings from using the credits were at least 20x what the migrations cost. We did the migration by having reverse proxies in each environment that could proxy to backends each place, set up a VPN between them,…

My estimates - the company had a combined total of $200,000 dollars in credits across both clouds and the OP charged $10,000 for the migrations.

Re: Ask HN: Have you ever switched cloud?

#195
post #40

Yes. I once did zero downtime migration first from AWS to Google, then from Google to Hetzner for a client. Mostly for cost reasons: they had a lot of free credits, and moved to Hetzner when they ran out. Their savings from using the credits were at least 20x what the migrations cost. We did the migration by having reverse proxies in each environment that could proxy to backends each place, set up a VPN between them,…

We haven't done a cloud migration, but I know from zero-downtime DATABASE migrations that if you do this: - don't use any cloud service that isn't a packaged version of an installable/usable OSS project - architect your services to be able to double-write and switchover the read source with A/B deployments If you can migrate your database without downtime this way, then you are much more flexible than if not.

You're right double-writes are flexible, and it's great when it works. With schema migrations I'm fine with it, because you can usually enforce consistency.

But with migrations from one database to another at different locations, I'm lukewarm to it because it means having to handle split-brain scenarios, and often that ends up resulting in a level of complexity that costs far more than it's worth. Of course your mileage may vary - there are situations where it's easy enough and/or where the value in doing it is high enough.

Re: Ask HN: Have you ever switched cloud?

#196
post #48

In my company, we were aware about the potential honeypots in each cloud and we developed our product from the first commit to be deployed on 3 (!) clouds: AWS, Azure, IBM. And while we made it work by sticking to the least common denominator which was FaaS/IaaS (Lambda, S3, API GW, K8s). It was certainly not easy. We also ignored tools that could've helped us greatly only against a single cloud in order to be multi…

Was it a business or technical decision to do multi-cloud?

Did you run simultaneously in 3 clouds? Can you explain the setup?

If not, did you do just run on each for a while to test, or have a reason to switch?

This is probably an impossible question to answer, but: were the savings/benefits of doing this actually worth the engineering costs involved in the end? Eg even if you chose what turned out to be the most expensive, worst option, would the company ultimately have been in a better place by having engineering focused on building things to increase customer value instead?

Re: Ask HN: Have you ever switched cloud?

#197
post #40

Yes. I once did zero downtime migration first from AWS to Google, then from Google to Hetzner for a client. Mostly for cost reasons: they had a lot of free credits, and moved to Hetzner when they ran out. Their savings from using the credits were at least 20x what the migrations cost. We did the migration by having reverse proxies in each environment that could proxy to backends each place, set up a VPN between them,…

My estimates - the company had a combined total of $200,000 dollars in credits across both clouds and the OP charged $10,000 for the migrations.

You're reasonably in the ballpark. It was complicated a bit because hosting on Hetzner was so much cheaper for them that $1 in credit was not worth $1 to them, as if/when they had to spend cash they spent substantially less than that at Hetzner.

Re: Ask HN: Have you ever switched cloud?

#198
post #14
post #5

Yes, switched from AWS to Google to Azure. Don't switch to Azure unless your employer forces you, you will regret it. Google is great on a technical level though, especially if you do things with Kubernetes.

> Don't switch to Azure unless your employer forces you, you will regret it. Can you give any details? Pricing, reliability, weird quirks you have to program around, ...?

The biggest problems: things don't work, things don't compose, and support is unknowledgeable & ineffective.

Things don't work: outages, bugs. So many I'm not sure where to start: ACR: slow as molasses. I don't know why they didn't build it on blobstore. Many queries are clearly O(images) even when they shouldn't be. Throughput it terrible and made worse by inane pagination; listing images, for example, has a throughput of something like 60kbps. And "b", as in bits. Minutes for 3MB of data. It's absurd; they think that's "working as expected". AKS: they manage the API server in AKS, and we find it is routinely non-responsive. We went through a quarter-long support ticket, which went back and forth between "you're putting too much load on the API server" -> "we don't think we are, what load is there?" "here's the top queries" and they're all queries from like, the cluster controllers — which are also managed by AKS/Azure. -> "well there's too much load!". App Gateway: normally stable, but had an outage when Let's Encrypts old root expired. (We were using a cross signed cert — i.e., our cert was valid, but App Gateway failed it, i.e., a false negative validation.) They never acknowledged the outage, and the support ticket we filed didn't get a response until like days later — missing support's SLA — meanwhile, some engineer somewhere clearly fixed the service, as it started working all on its own. ACR: we used to get 500s. IDK if these still happen or not, as we retry-looped most of the spots that were hitting them. Support response was ridiculous "what's your ISP?" "… this is inside your network, Azure, and my ISP wouldn't cause you to serve a 500." "Maybe the image is too big" "The image is ~100 KiB" … global AAD outages, the status page going down while the twitter account is like "check the status page", the Portal has issues (a few days ago listing subscriptions just returned 0 rows. Like, okay, I guess I'm not looking at those today), the activity log will occasionally just return errors, or today, it returned 0 entries where I knew there to be entries.

Composability: a cloud provider's job is to offer bricks from which I can build whatever infra my company's needs demand. But Azure constantly says "well, no, you can't put those bricks together that way." IPv6 anywhere in your vnet? You can't add a managed PostgreSQL server to that vnet, cause not only does it not support IPv6, you can't add it to an IPv4-only subnet in a dual-stack vnet. Like, the entire point of dual-stack…; also, when we attempted this, the API request took 2 hours to fail, and failed with "Internal error, please retry", which we then did, like chumps. 4 hours later, support ticket. AKS will add new features (that ought to have been there from day 1) like nodepools … but only to newly created cluster. You want to take advantage of that? Too bad! Recreate that cluster from scratch!

Support: Azure has no meaningful tooling for handling bugs in their services. The only hammer they have is support, and by god it's going to hit that nail. Support might (if you can get them to admit that yes, shit's broken) field a bug report, but then the support ticket is closed. Is the bug fixed? How will you know? IDK. Also, AFAICT, certain products you just can't open a support ticket for; notably, the portal. It isn't an "Azure service", so it isn't in the list of things to select from. Also, they override the mouse wheel on the list of services, so scrolling ~1 "detent" on a trackpad results in the list scrolling at Mach 4. Support tickets lack URLs, so they're unlinkable. Occasionally whatever the agents use to view info on tickets gets desynced from the ticket, and new replies in the portal are black-holed (but email replies still work). You can't put ">" in a support ticket, it's not allowed. You can't upload certain file formats, it's not allowed. (E.g., want to send video of a bug? Not today!) SLAs are regularly missed, and the responses often ask for information included in the opening ticket. And … the agent's grasp on English is frankly terrible. (I'd accuse them of out-sourcing it, but we once had an agent on a ticket go dark on us … because he was in Texas when Texas losts its power.)

Honorable mention: My God, AAD is trying to be as complex as can be. Apps, Service Principals (which are needlessly and confusingly just called "Enterprise Apps" in the UI! — oh and the search box for that page doesn't work), Roles, Permissions, Role Assignments, Tenant, Subscriptions … oh my God. Like, AWS IAM is frankly a terrible implementation, but Azure AAD makes AWS IAM look amazing.

My entire 2+ year experience with Azure has made me an ardent believer in AWS, and willing to try GCP.

I need to write this all in a blog post.

Re: Ask HN: Have you ever switched cloud?

#199
post #40

Yes. I once did zero downtime migration first from AWS to Google, then from Google to Hetzner for a client. Mostly for cost reasons: they had a lot of free credits, and moved to Hetzner when they ran out. Their savings from using the credits were at least 20x what the migrations cost. We did the migration by having reverse proxies in each environment that could proxy to backends each place, set up a VPN between them,…

Several years ago I had a string of clients who also came into enough Google Cloud credits to make a switch worthwhile. For these companies it wasn't a problem to have a few minutes of downtime, so the task was simply recreating their (usually AWS) production environment in Google Cloud.

If it had been a one off migration we might have done the same, but since the end goal was Hetzner from the start, and that meant their architecture needed to handle the HA piece anyway, and served as nice validation we really could fail over without anything going down.
Post reply on HN