Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Resizes 150M Images Every Day with Go and C++

blog.discordapp.com

Post reply on HN
Print view View original

61–70 of 157 posts

Re: How Discord Resizes 150M Images Every Day with Go and C++

#61

Post by malux85 »

AndrewStephens wrote: post #16
Xeoncross wrote: post #8

Why don't more companies resize images client-side first using and then save the server some work by only asking it to verify the result by - resizing to the same size - removing metadata This results in much faster transfer (10x less bandwidth used often for mobile uploads) and reduces server load by "farming" out the work to the clients. https://developer.mozilla.org/en-US/docs/Web/API/CanvasRende... # Edit: On Kee…

A few reasons: 1) Although the site serves up images at 1024 pixels (or whatever) today, in the future they may want larger images. When everyone is rocking 10K monitors and 6K phone displays, those small images are going to look pretty bad. 2) The original image has some metadata that they want to keep (geolocation, etc). 3) They think they can do a better and more consistent job resizing than the various browsers,…

agree on 3) most browsers just use linear interpolation when resizing images, which makes sense from a performance point of view, but looks terrible. Better to use a bi-linear or cubic resize, more computing up front, but better images, this is probably the reason they do it

Re: How Discord Resizes 150M Images Every Day with Go and C++

#63

Post by [deleted] »

sgk284 wrote: post #22

There is already an (unofficial Google) image proxy written in Go that is quite fast, does caching (local or backed by S3/GCS), and does other nice things like smart cropping: https://github.com/willnorris/imageproxy Seemed like a lot of unnecessary work for them to reimplement a service from scratch without gaining any major perf benefits over their existing one and without leaning on an existing well-known and well…

[deleted]

Re: How Discord Resizes 150M Images Every Day with Go and C++

#64

Post by SaltyBackendGuy »

AndrewStephens wrote: post #16
Xeoncross wrote: post #8

Why don't more companies resize images client-side first using and then save the server some work by only asking it to verify the result by - resizing to the same size - removing metadata This results in much faster transfer (10x less bandwidth used often for mobile uploads) and reduces server load by "farming" out the work to the clients. https://developer.mozilla.org/en-US/docs/Web/API/CanvasRende... # Edit: On Kee…

A few reasons: 1) Although the site serves up images at 1024 pixels (or whatever) today, in the future they may want larger images. When everyone is rocking 10K monitors and 6K phone displays, those small images are going to look pretty bad. 2) The original image has some metadata that they want to keep (geolocation, etc). 3) They think they can do a better and more consistent job resizing than the various browsers,…

> 2) The original image has some metadata that they want to keep (geolocation, etc).

Isn't exif data something you should strip out?

Re: How Discord Resizes 150M Images Every Day with Go and C++

#65

Post by jononor »

stmw wrote: post #9
throwthisawayt wrote: post #2

Did it seem to anyone else that sticking to Python would have been way easier? It didn’t seem like any of the performance gains were through Golang.

From reading this, seems HTTP handling speed was important to them? which Go is probably better for. Also, interfacing Python to C/C++ is pretty unpleasant.

In Python they already had an extremely fast library with bindings available.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#66

Post by Karrot_Kream »

fleitz wrote: post #57
smaili wrote: post #10

Earlier quoted context omitted.

I believe this little piece answers your question: > We likely could have addressed this behavior in Image Proxy, but we had been experimenting with using more Go, and it seemed like a good place to try Go out. At the heart of if, they were looking for opportunities to use more Go in their stack and they deemed this situation as a fit.

The age old solution in search of a problem.

I think that's a bit reductionist, no? There are many reasons they may have been searching for moving to Go. Off the top of my head I can think of:

1. Static typing increasing confidence and velocity

2. Better developer-facing tooling increasing velocity

3. More employees knowledgeable about Go than Python

4. More enthusiasm (and therefore faster velocity) around Go development.

The blog post was about the engineering challenges they faced and how they solved them and I think it was a great write-up in that regard. The post wasn't about why they switched this service from Python to Go.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#67

Post by caltrops »

Impossible wrote: post #55
bri3d wrote: post #15

Earlier quoted context omitted.

I'm not sure why this is being downvoted - image processing is one of the most dangerous parts of a common consumer-facing web software stack. By and large this is because image container formats are poorly documented, overly broad, and rely on a lot of tricky binary parsing that's easy to mess up in an unsafe programming language. It's also one of the most obvious ingress points for untrusted binary data uploaded by…

The downvotes are also because it's a somewhat cliche comment on HN now. Anytime anyone is doing any with C or C++ that is even indirectly web facing, "this could be unsafe!!!" is an obligatory comment, even though all major tech companies have core components written in C++, and there are big web apps that have been running for years that are mostly written in C or C++. Security is definitely a concern, but these ki…

This isn’t one of those. Handing large amounts of unvalidated user input to these libraries is particularly dangerous.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#68

Post by fleitz »

Karrot_Kream wrote: post #66
fleitz wrote: post #57

Earlier quoted context omitted.

The age old solution in search of a problem.

I think that's a bit reductionist, no? There are many reasons they may have been searching for moving to Go. Off the top of my head I can think of: 1. Static typing increasing confidence and velocity 2. Better developer-facing tooling increasing velocity 3. More employees knowledgeable about Go than Python 4. More enthusiasm (and therefore faster velocity) around Go development. The blog post was about the engineerin…

It might be, then again I see a lot of wheel reinvention in tech / NIH syndrome.

I'm the kind of hacker who if a service runs out of memory every 2 hours, writes a crontab to restart it every hour after X random minutes so they don't all restart at the same time. It gets a lot of eye rolls from the other engineers searching for perfection, but it tends to produce services quickly that are highly reliable.

And look now the engineers who like chaos monkey don't even have to set that up. It's built in.

It looks like most of the savings were in switching from pillow to opencv, something that thumbor already does. https://github.com/thumbor/opencv-engine

Re: How Discord Resizes 150M Images Every Day with Go and C++

#69

Post by TrinaryWorksToo »

Lutin wrote: post #49
Xeoncross wrote: post #8

Why don't more companies resize images client-side first using and then save the server some work by only asking it to verify the result by - resizing to the same size - removing metadata This results in much faster transfer (10x less bandwidth used often for mobile uploads) and reduces server load by "farming" out the work to the clients. https://developer.mozilla.org/en-US/docs/Web/API/CanvasRende... # Edit: On Kee…

This is for proxying images that users link in chat, not for when users upload images to the service. It doesn't make sense to talk about doing this resize on the client, as the client doesn't have the image.

That is a great point. It could still be feasible to cache the image on the client, and have it do the resize. Although I probably wouldn't accept it as a client, especially if my internet cuts out halfway through.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#70

Post by abiox »

caltrops wrote: post #44
GuB-42 wrote: post #42

Earlier quoted context omitted.

The downvote is probably because the comment implied that the issue is that the image processing is done in "unsafe" C++ and that another language should have been used. However, there isn't much choice. Performance is very important in image processing, so much that many libraries contain hand-written assembly. In the article, it says that 90% of processing power is dedicated to it. Using a safer language in a safe…

How much does a hack of all your data and/or a major outage cost? I also recommended a mitigation strategy for unsafe code. Complaining that security is too hard is the reason for the situation we find ourselves in as an industry.

> How much does a hack of all your data and/or a major outage cost?

seems to vary wildly. for some, it's not that expensive.

Post reply on HN