Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cost of serving billions of images per month

medium.com

Post reply on HN
Print view View original

51–60 of 138 posts

Re: Cost of serving billions of images per month

#51

Post by paulddraper »

scarface74 wrote: post #46
ron22 wrote: post #17

Earlier quoted context omitted.

They tell you in the blog post. "We continue to use Heroku as our main web platform. Despite its premium cost over AWS, Azure, and Google Cloud, Heroku’s built-in deployment and configuration tools allow our team to move faster, more confidently, and more reliably. As we’ve detailed previously, the alternatives would undoubtably be cheaper on paper. But in reality, the increased simplicity and freedom offered by Hero…

I’m far from a Devops guy. But deploying to AWS using their native tools are Azure using Microsoft’s tools is dead simple.

Yeah something doesn't add up.

I'll believe Heroku is easier, but can it be that much easier? For people spending piles of cash on their webscale?

Re: Cost of serving billions of images per month

#52

Post by sudhirj »

quickthrower2 wrote: post #22
quickthrower2 wrote: post #5

Earlier quoted context omitted.

Moreover what if Heroku goes out of business? Then they’re f’d. Hosting on k8s, using Terraform or whatever would give them a lot of automation and provider independence.

I’m guessing I’m being grayed because I’m suggesting Heroku might go out of business? Ok let me explain. That is one scenario, it’s unlikely but there might be other similar scenarios that pose risk being tied in to one providers stack. DDOS, TOS issues, simply out growing them, Heroku themselves having TOS issues with their cloud. There are almost weekly posts on HN of “this major service is down” and “that major se…

Heroku mandates the use or 12Factor apps, which makes it much easier to migrate. They support docker as well. Pretty much every app I write I usually starts on Heroku and migrate later if the client / traffic needs it, and it’s always been seamless. Most apps have Dev/test on Heroku with a staging and production on AWS, and because of the 12 factor principles, which also work on Beanstalk / ECS / normal servers it’s never a problem.

Re: Cost of serving billions of images per month

#53

Post by sudhirj »

paulddraper wrote: post #51
scarface74 wrote: post #46

Earlier quoted context omitted.

I’m far from a Devops guy. But deploying to AWS using their native tools are Azure using Microsoft’s tools is dead simple.

Yeah something doesn't add up. I'll believe Heroku is easier, but can it be that much easier? For people spending piles of cash on their webscale?

I do both at work, small sites on Heroku and big / compliance required ones on AWS, and the Heroku ones don’t need a devops team, while the AWS deployments do. Given the cost of building the devops teams and the time they spend, it is a big factor. Prefer to default to a Heroku unless there’s a clear cost benefit analysis that includes human and opportunity costs.

Re: Cost of serving billions of images per month

#54

Post by vincent_s »

amanzi wrote: post #11

So they must have some form of business model, right? All I can see on their website is that everything is for free, even their API which can be used for free without any limits or restrictions. And given the already huge Imgix costs, abusing or over-using their API to request and resize images would directly increase their overall costs. According to Crunchbase ( https://www.crunchbase.com/organization/unsplash#sect…

"He said it’s too soon to know exactly what that [business] model will be, but it will involve blockchain technology and cryptocurrency" [1]

[1] https://techcrunch.com/2018/02/15/unsplash-simple-token-seri...

Re: Cost of serving billions of images per month

#55

Post by judge2020 »

qes wrote: post #4
no1youknowz wrote: post #2

I like Unsplash. They and Pixabay are what I use for apps I have developed in the past. But simply, this floors me. I looked at their costs and with some developer muscle, you could find savings such as: - Move Fastly to Cloudflare. They don't expressly say what the cost is for that. But moving to CF would eliminate it. - Move Heroku to Digital Ocean. It's not difficult to create a fully redundant solution. - Move fr…

They say they spend $12,700/month on Fastly. I guess I don't know offhand the difference between Fastly and Imgix (which they also say is their CDN), but the levels of bandwidth they're talking about are not going to be free at Cloudflare. We serve significantly more bandwidth and requests than their 2016 numbers at under $15k compared to their $17.5k, and we're not particularly lean on infra costs being a .Net shop…

> the levels of bandwidth they're talking about are not going to be free at Cloudflare

Correct. Cloudflare's terms state that you can't serve a disproportionate/substantial amount of non-HTML content, so they would need to sign up for Enterprise with a full contract to get image serving via CF.

Enterprise is generally $5k+/month, but CF contracts can offer a subset of enterprise features from $500-$5k/month, so I would guess ~3k/month just to serve multiple petabytes of images. This doesn't eliminate the bandwidth costs so CF may not be the single solution for the bandwidth costs.

Re: Cost of serving billions of images per month

#56

Post by manigandham »

tpetry wrote: post #18
manigandham wrote: post #14

Earlier quoted context omitted.

And some projects really are just simple. Scale makes no difference to their services, it's a basic OLTP + search with resizing image proxy. This doesn't have any SLAs nor is it critical software either. 24/7 oncall is completely unnecessary.

Don‘t underestimate the massive cpu power needed for scaling images at their request rate

We run an ad platform that resizes, crops to the important section, transforms with text, and more with billions of requests per day. It's very cacheable, horizontally scalable, and CPU is cheap.

The main scaling challenge here is serving and bandwidth, both of which are handled by their CDN. I expect 5 figure costs for that but the rest of the platform is very expensive.

Re: Cost of serving billions of images per month

#57

Post by ma2rten »

HelloFellowDevs wrote: post #49
busymom0 wrote: post #13

Earlier quoted context omitted.

This reminds me of Soundcloud a few years ago when they were giving away API access for free for unlimited plays but then were almost close to bankrupt and pretty much shut down all API plays. I ended up shutting down my 3rd party app then but not sure what's their API limits now. Twitter recently made huge changes to their API limits too which impacted a lot of 3rd party apps. I hope Unsplash doesn't end up similar…

On the note of Twitter's restriction of API usage, Jack talked about it recently on the podcast Tales from the Crypt [0]. He doesn't give any specifics but attributes Twitters early struggle with uptime to unrestricted API usage [1]. [0] https://podcasts.apple.com/us/podcast/tales-from-the-crypt/i... [1] https://overcast.fm/+KiHpBJHGo/15:30

That seems silly to me. They could have easily served their api from another server.

Re: Cost of serving billions of images per month

#58

Post by Ayesh »

Wow, these are some serious expenses!

I run a service that does ~30 million requests a month, and I only spend $15 on it. It's an apples to orange comparison, but looking at their costs breakdown, I'm sure a couple engineers with a couple weeks can dramatically reduce their costs.

For me, the biggest cost savings were at browser caching. With `immutable` and far future cache expiration dates, webp images, etc, I could pull of the 30m and growing stats without the server load going more than 0.40.

Re: Cost of serving billions of images per month

#59

Post by quickthrower2 »

scarface74 wrote: post #45
kitotik wrote: post #36

Earlier quoted context omitted.

I find it much easier to maintain terraform code that allows me to use multiple cloud providers than if I had to do the same without terraform. You seem to have some insight to share here though, could you expound? *edit typo

How would Terraform help you move between cloud providers when all of the provisioners are specific to the cloud platform. If you’re a small startup, the least of your business risks are one of the major cloud providers shutting down.

> How would Terraform help you move between cloud providers when all of the provisioners are specific to the cloud platform

You don't move. You load balance across the clouds prior to the disaster.

You have your stuff work on both. I mentioned K8s because you could set up a managed cluster of that on a few cloud providers, and most of your TF will be the same in terms of setting up k8s, with some differences on how you set up those clouds.

This might be overkill for many people though so see below...

> If you’re a small startup, the least of your business risks are one of the major cloud providers shutting down.

I agree. a major cloud provider wont shut down....

... but they might shut YOU down.

Why? Billing Issues / TOS / 'Suspicious Activity' [0] / etc.

Now do you mitigate for that? Not necessarily but it is worth considering if you need to.

At the preparedness extreme you have a probe that detects the problem and flicks you over to cloud 2. Or a load balancer as I mention above (which would probe). That's probably too much for a scrappy startup.

But a middle ground is you have tech that is easy to move. Doesn't have to be k8s/TF but maybe a bash script you run on a new debian VM or whatever. Then you phone one of you awesome developers at 3am and tell them to migrate to AWS or whatever, and because it's easy they'll figure it out as they go, most stuff running by 3:30am and everything dandy by 5am.

The other extreme is you are tied in heavily to specific stacks by specific providers, and it take X hours/days to get back online again.

I'm not recommending to anyone what to do here - but I am saying consider the black swan events. You might consider them and say no I want my devs adding feature X so we can sell more. Fine, but I think when you can spend $100k a month on cloud you can probably afford to think about it a bit.

[0] Source: one of the major cloud providers cut all our services for 12 hours due to "suspicious activity". Turned out later it was due to a reused IP we were given from a pool that someone else f'd with. They gave us some credits to be nice afterwards.

Re: Cost of serving billions of images per month

#60

Post by ALittleLight »

vincent_s wrote: post #54
amanzi wrote: post #11

So they must have some form of business model, right? All I can see on their website is that everything is for free, even their API which can be used for free without any limits or restrictions. And given the already huge Imgix costs, abusing or over-using their API to request and resize images would directly increase their overall costs. According to Crunchbase ( https://www.crunchbase.com/organization/unsplash#sect…

"He said it’s too soon to know exactly what that [business] model will be, but it will involve blockchain technology and cryptocurrency" [1] [1] https://techcrunch.com/2018/02/15/unsplash-simple-token-seri...

Is there a way I can short this company?
Post reply on HN