Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Uploadcare Built a Stack That Handles 350M File API Requests per Day

stackshare.io

Post reply on HN
Print view View original

21–30 of 42 posts

Re: How Uploadcare Built a Stack That Handles 350M File API Requests per Day

#21

Post by notrheadagain »

lukb wrote: post #19

That's a great read. I've always been interested in learning how such tech-oriented companies found their initial traction. Are there any blog posts / articles / podcasts about Uploadcare's early days and the search for the product/market fit?

I just got wind of it, we at Uploadcare will soon be releasing an article with more info about the early days :) And, I believe, a podcast or two. Thanks for this question, btw. Would you elaborate on what you would like to know? It'll help us compile a great article, thanks :)

Re: How Uploadcare Built a Stack That Handles 350M File API Requests per Day

#23

Post by notrheadagain »

sigi45 wrote: post #20

By using AWS.

AWS is an infrastructure. It could be Google Cloud or something else. When writing the article, we wanted to convey how frameworks are interconnected and try and estimate the number of "Moving Parts" (I liked this one) :)

Re: How Uploadcare Built a Stack That Handles 350M File API Requests per Day

#24

Post by dmitrymukhin »

copyconstruct wrote: post #15

I wonder what's the breakdown between unique files delivered as opposed to files delivered from the CDN cache. Also, what's the breakdown for file uploads, manipulation and delivery? The 350M API requests per day would make more sense if we get this brakdown

Cached/uncached file delivery is close to the universal 80/20 ratio. Cached operations are not included in that number.

Unfortunately, I can't say anything more than that.

Re: How Uploadcare Built a Stack That Handles 350M File API Requests per Day

#25

Post by dmitrymukhin »

cagenut wrote: post #16
dmitrygr wrote: post #7

350M/day = just about 4K QPS. Is that considered impressive nowadays?

Impressive or not I just wish people would stop using monthly averages in the headline like this. You can't really make the case that this in-depth stack dive is for a layman audience, so you have to know what a meaningless metric it is.

I totally agree, but this was one of the requirements of the editor to have a "good marketable" headline. And we have to admit that this worked quite well.

On the other hand I feel that the article is very light and is indeed more for layman audience :) In depth one would be 10-15 times longer (and 100 times harder to write).

Re: How Uploadcare Built a Stack That Handles 350M File API Requests per Day

#27

Post by copyconstruct »

dmitrymukhin wrote: post #24
copyconstruct wrote: post #15

I wonder what's the breakdown between unique files delivered as opposed to files delivered from the CDN cache. Also, what's the breakdown for file uploads, manipulation and delivery? The 350M API requests per day would make more sense if we get this brakdown

Cached/uncached file delivery is close to the universal 80/20 ratio. Cached operations are not included in that number. Unfortunately, I can't say anything more than that.

Curious - does that mean you serve close to 1.75 billion requests per day, out of which 350M are unique requests that exercise your stack instead of being served from a CDN. It'd be interesting to know more about what's the number of transformations you do at peak, if you can talk about it.

Re: How Uploadcare Built a Stack That Handles 350M File API Requests per Day

#28

Post by koolba »

BillinghamJ wrote: post #13
nametube wrote: post #11

User generated content (especially images) are a great attack vector, what do you do to isolate/mitigate against attacks like that ?.

How so? Obviously they won’t be executing (or likely even analysing) any of the uploaded content. Similarly, browsers should not generally be particularly vulnerable to malicious content being loaded with appropriate MIME types in appropriate containers (e.g. ) It sounds like you should be asking how browsers protect users from malicious content. Perhaps you could elaborate?

You can perform a denial of service attack on a naive server with a maliciously crafted PNG. Just send a zip bomb and see what happens when it decompressed it. The naive approach will crash the server when it tries to malloc successively larger buffers.

https://www.bamsoftware.com/hacks/deflate.html

Re: How Uploadcare Built a Stack That Handles 350M File API Requests per Day

#29

Post by acdha »

nametube wrote: post #17
BillinghamJ wrote: post #13

Earlier quoted context omitted.

How so? Obviously they won’t be executing (or likely even analysing) any of the uploaded content. Similarly, browsers should not generally be particularly vulnerable to malicious content being loaded with appropriate MIME types in appropriate containers (e.g. ) It sounds like you should be asking how browsers protect users from malicious content. Perhaps you could elaborate?

Image and Video codecs come under attack quite often see https://blog.sucuri.net/2016/05/imagemagick-remote-command-e... In this context the image manipulation they do with pillow and the underlying libjpeg would be a potential source of vulnerabilities.

Significantly, it's not just libjpeg but every format supported by Pillow (http://pillow.readthedocs.io/en/3.4.x/handbook/image-file-fo...) — many of those vulnerabilities have historically been in obscure formats where the implementation has had far less attention than the mainline JPEG or PNG support.

Re: How Uploadcare Built a Stack That Handles 350M File API Requests per Day

#30

Post by lukb »

notrheadagain wrote: post #21
lukb wrote: post #19

That's a great read. I've always been interested in learning how such tech-oriented companies found their initial traction. Are there any blog posts / articles / podcasts about Uploadcare's early days and the search for the product/market fit?

I just got wind of it, we at Uploadcare will soon be releasing an article with more info about the early days :) And, I believe, a podcast or two. Thanks for this question, btw. Would you elaborate on what you would like to know? It'll help us compile a great article, thanks :)

Great to hear that!

The reason why it's particularly interesting to me, i.e. to someone with a dev background, is that the lean startup wisdom says you should be very specific about the customer you're after and Uploadcare seems like a solution targetted at a broad spectrum of customer segments. Of course, I'm happy to be proved wrong if there are one or two dominant customer segments that you address Uploadcare to. Also, you might have as well started out with a very specific customer persona and spread to other segments. Whatever it was, curious to know.

I guess many developers dream up products targetted at developers like them selves. Selling to fellow developers is hard. It would be great to read a success story for a change.

Post reply on HN