Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Canva saves Amazon S3 costs

canva.dev

Post reply on HN
Print view View original

71–80 of 181 posts

Re: How Canva saves Amazon S3 costs

#71

Post by tansan »

withinboredom wrote: post #51
jaza wrote: post #48

I'm really surprised how much anti cloud opinion is being voiced here (but maybe I should have known better, after all these years on HN). Even if you don't use any other cloud tech in your stack, my advice is, use S3 (or equivalent)! That level of durability, availability, and scalability, is not trivial to pull off, and is best made somebody else's problem. Even when you're as big as Canva.

I'm a big fan of Garage[1], which is a dead-simple S3 drop-in that you can host on your own drives. It's designed for consumer hardware with shitty internet in-between nodes. [1]: https://garagehq.deuxfleurs.fr/

Whoa! this is the first time I have seen deuxfleurs. Do french people use this over github?

Re: How Canva saves Amazon S3 costs

#72

Post by pc86 »

ttymck wrote: post #59
andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

Why do they need to get everything out? Some not-insignificant percentage of the content is surely archived/unused/unneeded assets. From customers who will never log in again or projects which are long over. Export the most recent 10% and put anything new on

I would think you could set up a migration process where existing things are loaded off of S3, and as soon as you make any changes, you save it to $NEW_STORAGE and delete it off of S3. Eliminates the need to egress anything, lowers your S3 bill over time, and as a user it'd be pretty hard to log in after a year and be upset that your FB cover image is no longer there, especially on the free tier.

Re: How Canva saves Amazon S3 costs

#73

Post by speed_spread »

andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

Nitpick: It's "you can _check out_ anytime you want but you can never leave"; the dissociation between checking out and actually leaving is what makes the line. Still a good reference for the S3 situation!

Re: How Canva saves Amazon S3 costs

#74

Post by pc86 »

notyourwork wrote: post #66
littlestymaar wrote: post #43

Earlier quoted context omitted.

Millions of dollars per year isn't exactly a trivial amount of money either… Nobody said it's some super easy thing one could do on their free time, but given the scale it makes sense to build the knowledge in-house instead of getting vendor-locked-in at the most expensive service provider on the market.

Id argue this isn’t true until your spend surpasses $100MM. Lower amounts don’t justify the extra headache of on-prem.

I know we're just arguing about amounts at this point but that's pushing $2M a week. It seems on on-prem would make financial sense long before that.

Re: How Canva saves Amazon S3 costs

#75

Post by withinboredom »

paxys wrote: post #69
withinboredom wrote: post #51

Earlier quoted context omitted.

I'm a big fan of Garage[1], which is a dead-simple S3 drop-in that you can host on your own drives. It's designed for consumer hardware with shitty internet in-between nodes. [1]: https://garagehq.deuxfleurs.fr/

All the marketing pages in the world mean nothing until they can show who currently uses them and at what scale.

Well, someone has to make that experiment and see if it’s right for them, right? It’s only a few years old, and most bigger companies are already sold on whatever they’re using. Smaller companies you’ve never heard of will be the first drivers, most likely.

FWIW, I have an EU region cluster (3 nodes per inner-region in the North, West and South). It’s all one logical region to S3 libraries and holds nearly half a terabyte.

It’s probably one of the most stable parts of my infrastructure. I don’t know (or need to know) what the replication lag is between regions, but IIRC, there’s some configuration of consistency.

Re: How Canva saves Amazon S3 costs

#76

Post by withinboredom »

tansan wrote: post #71
withinboredom wrote: post #51

Earlier quoted context omitted.

I'm a big fan of Garage[1], which is a dead-simple S3 drop-in that you can host on your own drives. It's designed for consumer hardware with shitty internet in-between nodes. [1]: https://garagehq.deuxfleurs.fr/

Whoa! this is the first time I have seen deuxfleurs. Do french people use this over github?

I believe deuxfleurs is a programming co-op. The UI is called gitea IIRC, an open source GitHub clone.

Re: How Canva saves Amazon S3 costs

#77

Post by adduc »

withinboredom wrote: post #76
tansan wrote: post #71

Earlier quoted context omitted.

Whoa! this is the first time I have seen deuxfleurs. Do french people use this over github?

I believe deuxfleurs is a programming co-op. The UI is called gitea IIRC, an open source GitHub clone.

This particular instance appears to be running Forgejo, a soft-fork of Gitea.

Re: How Canva saves Amazon S3 costs

#78

Post by jabart »

littlestymaar wrote: post #38
ramraj07 wrote: post #27

Earlier quoted context omitted.

Do you know how much it’ll cost to store 230 petabytes per month on premises at the same redundancy and availability level? I have done such calculations for academia at a smaller scale and unless you literally want it to last decades it’s definitely not worth it to do your own infra on this.

I keep seeing that argument, but you, like the others, don't provide the figures you claim you got from your calculation. Also, it seams that it's 230PB in total , not per month, which fits in an apartment-sized server room (you want to have several of such places, but it's not that big).

Wasabi has them. Scaling to 230pb would be more than their 1PB calculation. Taking their 5 yr total number, multiply to 230pb, divide by 5, then divide by 12 has a monthly total of 5,021,666. It's also 131 server racks according to wasabi. At 131 racks of storage you likely now need racks dedicated to networking since a top of rack switch wouldn't be enough. With space for the racks it's about 2,227sqft of space.

https://wasabi.com/blog/on-premises-vs-cloud-storage/

Re: How Canva saves Amazon S3 costs

#79

Post by electroly »

andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

Tip for anyone looking to spend seven-figure or more sums on one-time egress: Direct Connect egress is $0.02/GB. Rent a rack at a Direct Connect facility and get as many 10G fiber Direct Connects as you need, with corresponding flat rate 10G Internet ports with HE/Cogent/whatever transit provider. If you're going to be spending millions on egress, you could just hire someone to set this up for you. With that kind of spend you'd be crazy to pay the full $0.09/GB.

Edit: Note also that Snowball egress is $0.03/GB. Slightly higher egress, much lower setup cost. You'll have to do the math but they're both clearly attractive options vs. full price $0.09/GB egress.

Re: How Canva saves Amazon S3 costs

#80

Post by [deleted] »

onion2k wrote: post #6
andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

There's always a cheaper way to do something, but it's important to remember that 'lower price' often doesn't mean 'lower cost'. In order to get the lower price you need to spend on whatever the alternative option isn't doing in order to give that price saving. For example, moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops. And then yo…

[deleted]
Post reply on HN