Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Canva saves Amazon S3 costs

canva.dev

Post reply on HN
Print view View original

11–20 of 181 posts

Re: How Canva saves Amazon S3 costs

#11

Post by coldtea »

andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

If you still make a profit and grow with public Cloud, and you had to risk time-to-marker, reliability, man-years, to make something on your own, then it's not that crazy.

More of a "good problem to have" (and you can solve it when you need to).

Re: How Canva saves Amazon S3 costs

#12

Post by username_my1 »

onion2k wrote: post #6
andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

There's always a cheaper way to do something, but it's important to remember that 'lower price' often doesn't mean 'lower cost'. In order to get the lower price you need to spend on whatever the alternative option isn't doing in order to give that price saving. For example, moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops. And then yo…

usually you're right but the cost mentioned here is specific to BLOB data storage ... a very specific problem that can be isolated and run on external services and save the company lots of money.

Re: How Canva saves Amazon S3 costs

#13

Post by simonjgreen »

acchow wrote: post #5
rat9988 wrote: post #4

Real question, how to scale a business to petabytes of data without cloud? How is it done in practice?

10 Petabytes is 10,000 TB. That's only 454 spinning disk drives (22 TB each). Approximate this as 600 with Raid 5. You can easily fit this in 100 desktops. There's tons of OSS to manage all layers of this stuff now.

Slight tangent from the point, however: friends don't let friends use RAID5. Putting this out as a PSA reminder as it's not often discussed any more given the prevalance of cloud storage.

RAID5 gives you really the worst of all worlds. You get poor performance (n/4) due to the number of operations required and you get the fragility of a RAID storm that comes from a cascade RAID failure when a drive dies. A "stripe of mirrors" (RAID10) gives you the safest and the highest performance array (at scales where these decisions are being made which most people would see anyway). The common argument against is the cost (usableCapacity = totalCapacity/2) but my counter argument is that RAID5 is a timebomb with a secret countdown. RAID5 is great if you don't need the data on the array.

Source: >20 years running complex large arrays

Re: How Canva saves Amazon S3 costs

#14

Post by PaywallBuster »

onion2k wrote: post #6
andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

There's always a cheaper way to do something, but it's important to remember that 'lower price' often doesn't mean 'lower cost'. In order to get the lower price you need to spend on whatever the alternative option isn't doing in order to give that price saving. For example, moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops. And then yo…

or just consider different providers

and easily lower costs by 50% depending on balance storage/bandwidth

Re: How Canva saves Amazon S3 costs

#15

Post by simonjgreen »

ssss11 wrote: post #10
yashkadakia wrote: post #3

Earlier quoted context omitted.

Yup, we just moved large chunks of our SaaS platform from the cloud to dedicated servers with Kubernetes serving as an internal cloud. Ended up reducing our cost to 30% (70% saved!). We realized that instead of using scaling for peaks, by having additional dedicated servers available full time - it still worked out significantly cheaper. We also moved a lot of our internal processing to run in specific windows of tim…

Sorry what’s an “internal cloud”? Please tell me we haven’t renamed on-prem.

The term "private cloud" has existed for easily 15 years, pretty much straight after Google coined the term "cloud computing". The current deifnition of "on-prem" actually changed after cloud computing reached mass adoption. Prior to then, on-prem referred to "in your office" vs. "in a datacenter". These days "on-prem" means "not in public cloud".

Re: How Canva saves Amazon S3 costs

#16

Post by CogitoCogito »

username_my1 wrote: post #12
onion2k wrote: post #6

Earlier quoted context omitted.

There's always a cheaper way to do something, but it's important to remember that 'lower price' often doesn't mean 'lower cost'. In order to get the lower price you need to spend on whatever the alternative option isn't doing in order to give that price saving. For example, moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops. And then yo…

usually you're right but the cost mentioned here is specific to BLOB data storage ... a very specific problem that can be isolated and run on external services and save the company lots of money.

^^^

The cloud vs. on-prem argument often seems to ignore the (enormous) middle-ground. Just because one portion of your architecture would do well to be run outside the cloud, doesn't mean you take out all the other parts you don't want to deal with yourself. Furthmore, "on-prem" might mean in your building, in someone else's building co-located and you rent space and control it, or in someone else's building where they deal with most hardware, etc.

That said, maybe Canva has considered a non-AWS solution and decided against it. Or maybe they've gotten certain better deals from AWS. We can't really know for sure on the outside.

Re: How Canva saves Amazon S3 costs

#17

Post by zokier »

rat9988 wrote: post #4

Real question, how to scale a business to petabytes of data without cloud? How is it done in practice?

Quickly checking Dell sells you boxes that are advertised as 14PB per rack and have S3 api. You probably can do better by shopping around, but it gives a ballpark what is available in the on-prem market.

Re: How Canva saves Amazon S3 costs

#18

Post by Marazan »

ssss11 wrote: post #10
yashkadakia wrote: post #3

Earlier quoted context omitted.

Yup, we just moved large chunks of our SaaS platform from the cloud to dedicated servers with Kubernetes serving as an internal cloud. Ended up reducing our cost to 30% (70% saved!). We realized that instead of using scaling for peaks, by having additional dedicated servers available full time - it still worked out significantly cheaper. We also moved a lot of our internal processing to run in specific windows of tim…

Sorry what’s an “internal cloud”? Please tell me we haven’t renamed on-prem.

It will mean compute resources that can be transparently shared amongst different things (which is where the K8s comes in) as opposed to boxes dedicated to a single service.

Re: How Canva saves Amazon S3 costs

#19

Post by _flux »

ssss11 wrote: post #10
yashkadakia wrote: post #3

Earlier quoted context omitted.

Yup, we just moved large chunks of our SaaS platform from the cloud to dedicated servers with Kubernetes serving as an internal cloud. Ended up reducing our cost to 30% (70% saved!). We realized that instead of using scaling for peaks, by having additional dedicated servers available full time - it still worked out significantly cheaper. We also moved a lot of our internal processing to run in specific windows of tim…

Sorry what’s an “internal cloud”? Please tell me we haven’t renamed on-prem.

Yes, it's basically the same as "on prem", but instead just a bunch of individual (virtual) servers they are managed by a system that provides typical cloud APIs, such as OpenStack.

Re: How Canva saves Amazon S3 costs

#20

Post by andrewstuart »

onion2k wrote: post #6
andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

There's always a cheaper way to do something, but it's important to remember that 'lower price' often doesn't mean 'lower cost'. In order to get the lower price you need to spend on whatever the alternative option isn't doing in order to give that price saving. For example, moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops. And then yo…

>> moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops

This is the traditional “sell” for cloud computing and I don’t buy it at all.

The clouds would have you believe it doesn't make sense to run your own systems because you need too much specialist expertise to run your own systems.

The clouds sales pitch is the if you go cloud then you don’t need all these specialists.

That’s rubbish. Cloud operations need the same or more headcount if technical specialists, they’re just doing different things.

The old “don’t run your own systems, it’s cheaper and easier to go cloud” is just sales fiction.

Don’t believe it.

Post reply on HN