Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Canva saves Amazon S3 costs

canva.dev

Post reply on HN
Print view View original

81–90 of 181 posts

Re: How Canva saves Amazon S3 costs

#81

Post by fakedang »

electroly wrote: post #79
andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

Tip for anyone looking to spend seven-figure or more sums on one-time egress: Direct Connect egress is $0.02/GB. Rent a rack at a Direct Connect facility and get as many 10G fiber Direct Connects as you need, with corresponding flat rate 10G Internet ports with HE/Cogent/whatever transit provider. If you're going to be spending millions on egress, you could just hire someone to set this up for you. With that kind of…

Plot twist : Canva blogged this so someone could tell them some solution like this, without them having to hire the guy. :)

Re: How Canva saves Amazon S3 costs

#82

Post by amluto »

onion2k wrote: post #6
andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

There's always a cheaper way to do something, but it's important to remember that 'lower price' often doesn't mean 'lower cost'. In order to get the lower price you need to spend on whatever the alternative option isn't doing in order to give that price saving. For example, moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops. And then yo…

> And then you have to factor in things like the cost of downtime (on both sides of the equation, especially if you use us-east-1)

This is an important factor to remember when evaluating could costs. If you want to survive a cloud outage, you need a multi-AZ or multi-region deployment, and that costs developer hours. And you need to deal with the potentially catastrophic cost of inter-AZ or inter-region traffic, which can be catastrophic and/or cost more developer hours to mitigate.

Re: How Canva saves Amazon S3 costs

#83

Post by llanowarelves »

motoxpro wrote: post #42

A lot of people in this thread should start consultancies that transition companies off the cloud. Apparently, it would be easy for them to save tens of millions, in turn making millions for the consultant.

The thing about economics is that it's physics, with a time delay. We are seeing the time-delayed reaction to over-extension into the cloud play out, not instantly but in the downturn cycle.

Re: How Canva saves Amazon S3 costs

#84

Post by amluto »

maccard wrote: post #46
CogitoCogito wrote: post #16

Earlier quoted context omitted.

^^^ The cloud vs. on-prem argument often seems to ignore the (enormous) middle-ground. Just because one portion of your architecture would do well to be run outside the cloud, doesn't mean you take out all the other parts you don't want to deal with yourself. Furthmore, "on-prem" might mean in your building, in someone else's building co-located and you rent space and control it, or in someone else's building where t…

My team uses AWS for _everything_. There's three reasons for this on our team (and I've researched it for our use cases. - Consistency. Instead of saying "Oh look in AWS for this app, Azure for this one, and hetzner for this app, except it's test env is in AWS", it all just lives in AWS. It massively simplifies docs, onboarding, and reduces the amount of one-person specialised knowledge. - Engineering Costs. Similar…

> - Vendor Lock In. Yeah, it's there. If we have a service that uses data from S3, there's egress costs from S3 to , but not with EC2. We've consciously accepted this lock in for the time being.

This is where I think the FCC should take action.

To the extent that this issue is a mutually agreeable arrangement between you and Amazon, it seems obnoxious but does not seem like it rises to the level where regulators should take action. But it affects third parties too: specifically, it prevents non-AWS-hosted vendors from effectively marketing their services to you. In that regard, I think the FTC should try to put a stop to this. AWS should not be permitted to effectively subsidize its and its partners’ services over outside competitors.

(And the US Government should never have accepted cloud deals with excessive egress costs. Part of the bidding process should have been a requirement for networking outside the winning provider to be priced competitively with internal networking)

Re: How Canva saves Amazon S3 costs

#85

Post by xwowsersx »

andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

It's hard to imagine why they would ever need to get everything out. That would be extremely wasteful. If they wanted to move off of S3, they'd rely on the same type of analysis they present in this post + perhaps creating a mechanism whereby they permanently move an asset off of S3 if/when accessed.

Re: How Canva saves Amazon S3 costs

#86

Post by willcipriano »

chii wrote: post #54
onion2k wrote: post #25

Earlier quoted context omitted.

Don’t believe it. You don't need to 'believe'. You need to do an analysis of what you need and how much each option will cost. Then you can know . If your argument is "I believe onprem saves money" or "I bet AWS is cheaper" or "Jim Morrison came to me in a dream and said I should use Azure" then you haven't done enough research.

> you haven't done enough research. the research isn't free either. The more indepth and time it takes to do such research, the slower you come to a decision and ship. Cloud allows you to ship fast. It allows you to go without research - just accept the marketing, and pay up. You pay above-cost (compared to on-prem) when you grow to a certain size. But this is usually a worthy trade off tbh.

It's a good way to bypass legacy IT teams. A onprem server will have a bunch of snake oil endpoint protection products running on the box, bespoke config changes and take a couple months to get up and running, a ECS container is up and running in minutes and has whatever you shipped in the container without all the commentary from the peanut gallery.

Re: How Canva saves Amazon S3 costs

#87

Post by hinkley »

jabart wrote: post #78
littlestymaar wrote: post #38

Earlier quoted context omitted.

I keep seeing that argument, but you, like the others, don't provide the figures you claim you got from your calculation. Also, it seams that it's 230PB in total , not per month, which fits in an apartment-sized server room (you want to have several of such places, but it's not that big).

Wasabi has them. Scaling to 230pb would be more than their 1PB calculation. Taking their 5 yr total number, multiply to 230pb, divide by 5, then divide by 12 has a monthly total of 5,021,666. It's also 131 server racks according to wasabi. At 131 racks of storage you likely now need racks dedicated to networking since a top of rack switch wouldn't be enough. With space for the racks it's about 2,227sqft of space. htt…

140 racks is a quite a lot of tonnage of AC as well. And probably hot and cold rows to make it function.

Re: How Canva saves Amazon S3 costs

#88

Post by hinkley »

llanowarelves wrote: post #83
motoxpro wrote: post #42

A lot of people in this thread should start consultancies that transition companies off the cloud. Apparently, it would be easy for them to save tens of millions, in turn making millions for the consultant.

The thing about economics is that it's physics, with a time delay. We are seeing the time-delayed reaction to over-extension into the cloud play out, not instantly but in the downturn cycle.

So do you think server hardware companies will do well during a downturn?

Re: How Canva saves Amazon S3 costs

#89

Post by ignoramous »

xwowsersx wrote: post #85
andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

It's hard to imagine why they would ever need to get everything out. That would be extremely wasteful. If they wanted to move off of S3, they'd rely on the same type of analysis they present in this post + perhaps creating a mechanism whereby they permanently move an asset off of S3 if/when accessed.

> It's hard to imagine why they would ever need to get everything out. That would be extremely wasteful.

See the Docker x Cloudflare case study where improving cache-hit ratio by 2% (by moving to R2) decreased S3 egress fee by 66%: https://www.cloudflare.com/en/case-studies/docker/

Re: How Canva saves Amazon S3 costs

#90

Post by m1keil »

willcipriano wrote: post #86
chii wrote: post #54

Earlier quoted context omitted.

> you haven't done enough research. the research isn't free either. The more indepth and time it takes to do such research, the slower you come to a decision and ship. Cloud allows you to ship fast. It allows you to go without research - just accept the marketing, and pay up. You pay above-cost (compared to on-prem) when you grow to a certain size. But this is usually a worthy trade off tbh.

It's a good way to bypass legacy IT teams. A onprem server will have a bunch of snake oil endpoint protection products running on the box, bespoke config changes and take a couple months to get up and running, a ECS container is up and running in minutes and has whatever you shipped in the container without all the commentary from the peanut gallery.

A poor DX can happen in the cloud as well. Waiting weeks for IAM configuration to be solved or a security group to be opened..
Post reply on HN