Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Canva saves Amazon S3 costs

canva.dev

Post reply on HN
Print view View original

141–150 of 181 posts

Re: How Canva saves Amazon S3 costs

#141

Post by littlestymaar »

jabart wrote: post #103
hinkley wrote: post #87

Earlier quoted context omitted.

140 racks is a quite a lot of tonnage of AC as well. And probably hot and cold rows to make it function.

Hot/cold was factored in not the AC or power needs. I don't know many places you could just casually ask for 100+ racks. Maybe our midwest datacenters are just smaller than what is on the coast.

> I don't know many places you could just casually ask for 100+ racks.

That's obviously not something you can do on a whim, but at the same time we're talking about a seven figures investment plus the time to build the in-house team to manage this, so it's not going to happen overnight anyway.

Re: How Canva saves Amazon S3 costs

#142

Post by Dunedan »

Something which is often overlooked is that transitioning objects in S3 to another storage class using lifecycle policies is a one-way operation. If you later notice that the chosen storage class doesn't fit or your use case changes, you can only go "forward" using lifecycle policies and not "back" to e.g. "S3 Standard" [1]. Going "back" is therefore a quite costly and cumbersome operation, as you have to apply the change of the storage class to each object individually.

[1]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecy...

Re: How Canva saves Amazon S3 costs

#143

Post by littlestymaar »

shawabawa3 wrote: post #129
pc86 wrote: post #74

Earlier quoted context omitted.

I know we're just arguing about amounts at this point but that's pushing $2M a week . It seems on on-prem would make financial sense long before that.

Depends on your requirements. If you need 11 nines durability S3 is probably still cheaper

I really hope that Canva's executive dont believe their product need such garantees…

Re: How Canva saves Amazon S3 costs

#144

Post by AntonCTO »

withinboredom wrote: post #51
jaza wrote: post #48

I'm really surprised how much anti cloud opinion is being voiced here (but maybe I should have known better, after all these years on HN). Even if you don't use any other cloud tech in your stack, my advice is, use S3 (or equivalent)! That level of durability, availability, and scalability, is not trivial to pull off, and is best made somebody else's problem. Even when you're as big as Canva.

I'm a big fan of Garage[1], which is a dead-simple S3 drop-in that you can host on your own drives. It's designed for consumer hardware with shitty internet in-between nodes. [1]: https://garagehq.deuxfleurs.fr/

Thank you, that's amazing! While there isn't an extensive comparison available, I have come across a small benchmark that compares a few features. You can find it here: https://garagehq.deuxfleurs.fr/blog/2022-perf/

Re: How Canva saves Amazon S3 costs

#145

Post by grogenaut »

tough wrote: post #92
fakedang wrote: post #81

Earlier quoted context omitted.

Plot twist : Canva blogged this so someone could tell them some solution like this, without them having to hire the guy. :)

There's this law on the internet or SO or whatever where if you put the wrong answer in a post. Someone else in the comments will correct you with the correct answer. This in effect.

This isn't just the internet. I've used this at work several times to break stalemates. I've come into many stalled projects where people are arguing over trivial details or blue skying on requirements. 1-2 hours and a 7 page design doc later I'll have everyone review what I know is an 80% answer. Pretty much every time they'll all start attacking my design, pointing out minor issues and then I have them. They've agreed to my overall design and are into details.

Same if someone is calling out all of the general problems with solving a problem but not providing answers where you can't get them to bite.

Re: How Canva saves Amazon S3 costs

#146

Post by leon1717 »

lbotos wrote: post #114
tough wrote: post #92

Earlier quoted context omitted.

There's this law on the internet or SO or whatever where if you put the wrong answer in a post. Someone else in the comments will correct you with the correct answer. This in effect.

https://en.wikipedia.org/wiki/Ward_Cunningham#%22Cunningham'...

Cunningham is cunning!!!

Re: How Canva saves Amazon S3 costs

#147

Post by AntonCTO »

simonjgreen wrote: post #13
acchow wrote: post #5

Earlier quoted context omitted.

10 Petabytes is 10,000 TB. That's only 454 spinning disk drives (22 TB each). Approximate this as 600 with Raid 5. You can easily fit this in 100 desktops. There's tons of OSS to manage all layers of this stuff now.

Slight tangent from the point, however: friends don't let friends use RAID5. Putting this out as a PSA reminder as it's not often discussed any more given the prevalance of cloud storage. RAID5 gives you really the worst of all worlds. You get poor performance (n/4) due to the number of operations required and you get the fragility of a RAID storm that comes from a cascade RAID failure when a drive dies. A "stripe of…

RAID10 for 22TB disks with only two drives at "1" level? I would expect no less than 3 drives at such capacity. Otherwise there is a huge chance to lose the second drive while 1. changing the disc (=time, could be even days) and 2. replication.

What are your thoughts on implementing RAID 0 over four RAID 1 arrays, each consisting of three disks, resulting in a total of 12 drives?

Re: How Canva saves Amazon S3 costs

#148

Post by littlestymaar »

jabart wrote: post #134
littlestymaar wrote: post #132

Earlier quoted context omitted.

This is a marketing pitch for a cloud storage company and it shows: > - System hardware: $500,000 For one PB? Gold plated HDDs and cases I guess… > Assume 15% annual system maintenance over five years Do not run your hard drive cluster on the same floor as vibration-inducing machines, folks. /s According to Backblaze, even a 8 year-old hard drive is still twice more reliable than this, and 15% failure rate is what to…

Can you fit 10pb in a rack, sure if you want it all flash. You also need double that for mirroring, or quadruple that for raid 10. Double that if you want a mirrored copy of that array too. If you don't have high performance needs you can get by with slower hard drives. You also have the 25gb,40gb, or 100gb top of rack switch. Well you want two of those for redundancy. It starts to quickly add up. Annual system maint…

> Number 1 reason I'm in a datacenter is to swap a dead drive.

No doubt about that, when you have tens of thousands of drive you end up replacing some of them all the time, but not 15% of them every year…

> Oct 2020, StorONE announced a 1PB All-Flash array for $499k.

With Optane and all, right? The kind of things that's a massive overkill if your workload allows you to use Glacier…

> Other examples which are close to their pricing.

In that list, the only offers that are close to this pricing (but still almost 20% cheaper) also includes several years of support (which you probably don't want if you're Canva's scale).

Re: How Canva saves Amazon S3 costs

#150

Post by encodedrose »

If you're at a point of spending $1.6M to transition objects from one class to another and spending millions per month in storage costs -- you need to have a real conversation about if storing your data in a vendor-locked cloud is the right path forward. S3 isn't the only option, MinIO + dense storage is one viable option if your spend is high enough to justify running MinIO. Backblaze is another.

List price of Backblaze:

$0.005/GB - Base Storage

$0.01/GB - Egress (Use a data partner to bring this down to $0)

$0 - PUT API Requests

$0.004/10,000 - GET API Requests

No delete penalties a/k/a minimum storage

List price of Glacier Instant Retrieval:

$0.004/GB - Base Storage

$0.09/GB - Egress

$0.02/1,000 - PUT API Requests

$0.01/1,000 - GET API Requests

Billed for 90 days of storage

Post reply on HN