Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Canva saves Amazon S3 costs

canva.dev

Post reply on HN
Print view View original

101–110 of 181 posts

Re: How Canva saves Amazon S3 costs

#101

Post by betaby »

shadowtree wrote: post #96
andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

Most businesses treat cloud like a utility, not tech. Don't want to generate their own power, deal with sewage ... and compute. Which in 99.9% of the cases is the right decision. Sorry for all you infra techies, but that's the way tech matures.

Infra techies are fine. Likewise 'grid is mature' and yet most of buildings with floor space over NN m2 have team of electricians, being them direct hire or contractors. Same for the 'cloud' - 90%+ of the world deployed 'compute' is not cloud.

Re: How Canva saves Amazon S3 costs

#102

Post by dehrmann »

electroly wrote: post #79
andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

Tip for anyone looking to spend seven-figure or more sums on one-time egress: Direct Connect egress is $0.02/GB. Rent a rack at a Direct Connect facility and get as many 10G fiber Direct Connects as you need, with corresponding flat rate 10G Internet ports with HE/Cogent/whatever transit provider. If you're going to be spending millions on egress, you could just hire someone to set this up for you. With that kind of…

I worked at a place that did an AWS to GCP migration. I didn't look into the details, but they used a provider that basically gave them a fat, dedicated fiber link between the two (they picked the same GCP region). It was good enough that requests could have data cross clouds. It was expensive to set up, but it saved on egress costs and opened up options for migrating services.

Re: How Canva saves Amazon S3 costs

#103

Post by jabart »

hinkley wrote: post #87
jabart wrote: post #78

Earlier quoted context omitted.

Wasabi has them. Scaling to 230pb would be more than their 1PB calculation. Taking their 5 yr total number, multiply to 230pb, divide by 5, then divide by 12 has a monthly total of 5,021,666. It's also 131 server racks according to wasabi. At 131 racks of storage you likely now need racks dedicated to networking since a top of rack switch wouldn't be enough. With space for the racks it's about 2,227sqft of space. htt…

140 racks is a quite a lot of tonnage of AC as well. And probably hot and cold rows to make it function.

Hot/cold was factored in not the AC or power needs. I don't know many places you could just casually ask for 100+ racks. Maybe our midwest datacenters are just smaller than what is on the coast.

Re: How Canva saves Amazon S3 costs

#104

Post by Solvency »

orangepanda wrote: post #29

Offtopic — is it some cardinal sin to include a link to the home page from a documentation? I dont know what canva does, if I stumble on a documentation page first, should I not be allowed to see the product's landing page at some point? This is so prevalent, it has to be a conscious choice.

My favorite is when you click on some landing page made by a company.... and there is zero way to get to the rest of the site. Marketing-centric landing pages often completely strip out their global navigation for no reason.

Re: How Canva saves Amazon S3 costs

#105

Post by lowbloodsugar »

Solvency wrote: post #100
StratusBen wrote: post #98

This is an interesting post...though I'm surprised to see this kind of optimization occur so late stage/as large as Canva. There's quite a few anti-cloud opinions in the comments but for anyone who is hosted on S3 and would like this same level of automated analysis for cost savings on S3 (and other AWS services), we basically profile for this out-of-the-box on Vantage and it's free to get an optimization check to se…

"I'm surprised to see this kind of optimization occur so late" Every single day I read the trope on HN that "premature optimization is the root of all evil". So, these guys literally followed HN's creed.

Canva is ten years old. They grew to 1 million users in two years. Doing this work 8 years ago wouldn't have been "premature optimization". More likely, this is happening now simply because the financial environment has changed and companies are now giving increased weight to projects that save money over those that add features.

Re: How Canva saves Amazon S3 costs

#106

Post by dehrmann »

andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

The real story isn't how expensive it is to leave, it's how easy AWS makes it to spend. It used to be that you'd see how many servers a project needs and have to beg someone for them, and then only get half of what you wanted. With AWS, it's much easier to launch something new, but there isn't the same amount of scrutiny on spend.

Re: How Canva saves Amazon S3 costs

#107

Post by betaby »

oefrha wrote: post #40
jokethrowaway wrote: post #21

Canva is a case where AWS made sense for the founders. The founders anyway made billions and who cares if you give a few hundreds millions to Jeff and have to hire expensive AWS consultants if you can avoid hiring a few ops people that actually know how to operate a server /s More seriously, I think the sweet spot is to buy 2x o 3x what you expect to need but do it off the cloud. At Canva's scale you'll spend hundred…

Apple uses AWS for most(?) of their cloud services. TikTok grew to the #1 spot entirely on a public cloud (Alibaba). I’m sure they sure have hired “a few ops people that actually know how to operate a server” instead.

Apple operates their own data centers. Apple uses GCP as part of the Google's deal related to search.

Re: How Canva saves Amazon S3 costs

#108

Post by betaby »

nemothekid wrote: post #91
oefrha wrote: post #40

Earlier quoted context omitted.

Apple uses AWS for most(?) of their cloud services. TikTok grew to the #1 spot entirely on a public cloud (Alibaba). I’m sure they sure have hired “a few ops people that actually know how to operate a server” instead.

The anti-cloud sentiment on HN is as old as time; there will always be posters who can claim that they can build X with just dedicated machines on Hetzner for 1/10th of the cost. Yet AWS keeps growing and the people running these companies must be unable to do basic math. My account just crossed 11 years, and I think the whole cloud vs on-prem conversation is probably one of the most least productive conversations on…

> there will always be posters who can claim that they can build X with just dedicated machines on Hetzner for 1/10th of the cost

and some of them just did. Both know names and small shops. Big names like cloud-flare R2 or Backblaze b2.

Re: How Canva saves Amazon S3 costs

#109

Post by icedchai »

colechristensen wrote: post #9
acchow wrote: post #5

Earlier quoted context omitted.

10 Petabytes is 10,000 TB. That's only 454 spinning disk drives (22 TB each). Approximate this as 600 with Raid 5. You can easily fit this in 100 desktops. There's tons of OSS to manage all layers of this stuff now.

You can have a few petabytes in a single rack. 50-ish drives in a single 4U server is done.

What about single points of failure? Replication? High availability? Disaster recovery? Backup / snapshots/ versioning? This is one of the benefits of the cloud.

Re: How Canva saves Amazon S3 costs

#110

Post by electroly »

dehrmann wrote: post #102
electroly wrote: post #79

Earlier quoted context omitted.

Tip for anyone looking to spend seven-figure or more sums on one-time egress: Direct Connect egress is $0.02/GB. Rent a rack at a Direct Connect facility and get as many 10G fiber Direct Connects as you need, with corresponding flat rate 10G Internet ports with HE/Cogent/whatever transit provider. If you're going to be spending millions on egress, you could just hire someone to set this up for you. With that kind of…

I worked at a place that did an AWS to GCP migration. I didn't look into the details, but they used a provider that basically gave them a fat, dedicated fiber link between the two (they picked the same GCP region). It was good enough that requests could have data cross clouds. It was expensive to set up, but it saved on egress costs and opened up options for migrating services.

I believe the relevant GCP product is "Dedicated Interconnect" and for Azure it's "ExpressRoute Direct" but don't quote me on that; I only know AWS. In my example you can swap out the Internet port for one of those cloud interconnects and go straight from AWS to your new cloud. That's certainly what that provider was doing. I bet there's lots of good money in facilitating those migrations using this relatively simple technique.
Post reply on HN