Did it seem to anyone else that sticking to Python would have been way easier? It didn’t seem like any of the performance gains were through Golang.
[deleted]
How Discord Resizes 150M Images Every Day with Go and C++
11–20 of 157 posts
Re: How Discord Resizes 150M Images Every Day with Go and C++
#12Did it seem to anyone else that sticking to Python would have been way easier? It didn’t seem like any of the performance gains were through Golang.
I understand this is a personal preference, but having spent a good amount time with both Python and Go, FWIW I would also choose Go if I were solving the same problem.
Re: How Discord Resizes 150M Images Every Day with Go and C++
#13Why don't more companies resize images client-side first using and then save the server some work by only asking it to verify the result by - resizing to the same size - removing metadata This results in much faster transfer (10x less bandwidth used often for mobile uploads) and reduces server load by "farming" out the work to the clients. https://developer.mozilla.org/en-US/docs/Web/API/CanvasRende... # Edit: On Kee…
Because then you don't get to build out fun infrastructure like this and write it up in your company blog.
Re: How Discord Resizes 150M Images Every Day with Go and C++
#14Anybody knows how well libvips https://github.com/DAddYE/vips compares to liliput performance wise?
vips (Go binding) is included in the benchmarks mentioned in the post, but at the time of running them (~10 months ago) vips pulled 51482954 ns/op on a 1024x1024 test image, where as pillow-simd managed 3324135.3035 ns/op.
Re: How Discord Resizes 150M Images Every Day with Go and C++
#15I’d be very worried about a security issue with the unsafe C++ code. You really have to run this kind of complex parsing in a disposable containerized environment to do it safely. Or do everything carefully and in a memory safe language.
I'm not sure why this is being downvoted - image processing is one of the most dangerous parts of a common consumer-facing web software stack. By and large this is because image container formats are poorly documented, overly broad, and rely on a lot of tricky binary parsing that's easy to mess up in an unsafe programming language. It's also one of the most obvious ingress points for untrusted binary data uploaded by an end-user, which is always going to be dangerous.
See the persistent, years-long trend where mobile devices and game consoles get exploited via some combination of libtiff and libpng.
Re: How Discord Resizes 150M Images Every Day with Go and C++
#16Why don't more companies resize images client-side first using and then save the server some work by only asking it to verify the result by - resizing to the same size - removing metadata This results in much faster transfer (10x less bandwidth used often for mobile uploads) and reduces server load by "farming" out the work to the clients. https://developer.mozilla.org/en-US/docs/Web/API/CanvasRende... # Edit: On Kee…
A few reasons:
1) Although the site serves up images at 1024 pixels (or whatever) today, in the future they may want larger images. When everyone is rocking 10K monitors and 6K phone displays, those small images are going to look pretty bad.
2) The original image has some metadata that they want to keep (geolocation, etc).
3) They think they can do a better and more consistent job resizing than the various browsers, which is probably true.
Re: How Discord Resizes 150M Images Every Day with Go and C++
#17Why don't more companies resize images client-side first using and then save the server some work by only asking it to verify the result by - resizing to the same size - removing metadata This results in much faster transfer (10x less bandwidth used often for mobile uploads) and reduces server load by "farming" out the work to the clients. https://developer.mozilla.org/en-US/docs/Web/API/CanvasRende... # Edit: On Kee…
Because then you don't get to build out fun infrastructure like this and write it up in your company blog.
Sure you do! Don't you know that adding "Free" to a title increases the ROI by 40%?
"How Discord Resizes 150M Images Every Day for Free"
Re: How Discord Resizes 150M Images Every Day with Go and C++
#18Did it seem to anyone else that sticking to Python would have been way easier? It didn’t seem like any of the performance gains were through Golang.
I believe this little piece answers your question: > We likely could have addressed this behavior in Image Proxy, but we had been experimenting with using more Go, and it seemed like a good place to try Go out. At the heart of if, they were looking for opportunities to use more Go in their stack and they deemed this situation as a fit.
And they ended up open-sourcing the library they built, so it's a win on all sides.
Re: How Discord Resizes 150M Images Every Day with Go and C++
#19Why don't more companies resize images client-side first using and then save the server some work by only asking it to verify the result by - resizing to the same size - removing metadata This results in much faster transfer (10x less bandwidth used often for mobile uploads) and reduces server load by "farming" out the work to the clients. https://developer.mozilla.org/en-US/docs/Web/API/CanvasRende... # Edit: On Kee…
It may work in certain cases.
But it also means you need to know how you will display when you save it. Layout changes, screens change, how do you anticipate the future dimensions / resolution you will need out of the original?
Re: How Discord Resizes 150M Images Every Day with Go and C++
#20Why don't more companies resize images client-side first using and then save the server some work by only asking it to verify the result by - resizing to the same size - removing metadata This results in much faster transfer (10x less bandwidth used often for mobile uploads) and reduces server load by "farming" out the work to the clients. https://developer.mozilla.org/en-US/docs/Web/API/CanvasRende... # Edit: On Kee…
As mentioned in the post, one of our core product features is preventing your IP from being shared. Given that requirement, images shared in chat have to be proxied through our infrastructure. When doing this we save a lot of money and improve client performance by reducing image sizes.