Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Resizes 150M Images Every Day with Go and C++

blog.discordapp.com

Post reply on HN
Print view View original

91–100 of 157 posts

Re: How Discord Resizes 150M Images Every Day with Go and C++

#91

Post by amelius »

malux85 wrote: post #61
AndrewStephens wrote: post #16

Earlier quoted context omitted.

A few reasons: 1) Although the site serves up images at 1024 pixels (or whatever) today, in the future they may want larger images. When everyone is rocking 10K monitors and 6K phone displays, those small images are going to look pretty bad. 2) The original image has some metadata that they want to keep (geolocation, etc). 3) They think they can do a better and more consistent job resizing than the various browsers,…

agree on 3) most browsers just use linear interpolation when resizing images, which makes sense from a performance point of view, but looks terrible. Better to use a bi-linear or cubic resize, more computing up front, but better images, this is probably the reason they do it

But soon you can do any type of resizing through WASM on the browser.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#92

Post by mercwear »

manigandham wrote: post #74

Nice, but why? https://cloudinary.com , https://www.imgix.com , or https://www.filestack.com already exist and are well worth it for 99% of apps. Even at scale, it really doesn't cost that much to have someone else do it. You can use a thin proxy through your existing CDN if you want to save on their bandwidth fees. Also http://thumbor.org and https://imageresizing.net if you want a library to host yourself which are…

I agree. Offloading this type of work to a third party who does it really well is a smart move. Why manage additional code when it's not even core to what you do?

Re: How Discord Resizes 150M Images Every Day with Go and C++

#94

Post by Moru »

AndrewStephens wrote: post #16
Xeoncross wrote: post #8

Why don't more companies resize images client-side first using and then save the server some work by only asking it to verify the result by - resizing to the same size - removing metadata This results in much faster transfer (10x less bandwidth used often for mobile uploads) and reduces server load by "farming" out the work to the clients. https://developer.mozilla.org/en-US/docs/Web/API/CanvasRende... # Edit: On Kee…

A few reasons: 1) Although the site serves up images at 1024 pixels (or whatever) today, in the future they may want larger images. When everyone is rocking 10K monitors and 6K phone displays, those small images are going to look pretty bad. 2) The original image has some metadata that they want to keep (geolocation, etc). 3) They think they can do a better and more consistent job resizing than the various browsers,…

Our site would have been happy with full res images from the start. As it is now we are stuck with 80x80 images that needs replacing with higher res images since the originals was not kept in any sorted order.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#95

Post by manigandham »

reificator wrote: post #87
manigandham wrote: post #83

Earlier quoted context omitted.

It's images... seems like a very low risk situation, especially when they are served from a CDN.

As a user, the order of importance for Discord services is: * Voice * Text * Previews (images, gifs, and videos) Previews going down would be a pretty big deal for my communities based on the way we use the platform.

It’s just resizing, you still have the source images and can use those.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#96

Post by Xeoncross »

Moru wrote: post #94
AndrewStephens wrote: post #16

Earlier quoted context omitted.

A few reasons: 1) Although the site serves up images at 1024 pixels (or whatever) today, in the future they may want larger images. When everyone is rocking 10K monitors and 6K phone displays, those small images are going to look pretty bad. 2) The original image has some metadata that they want to keep (geolocation, etc). 3) They think they can do a better and more consistent job resizing than the various browsers,…

Our site would have been happy with full res images from the start. As it is now we are stuck with 80x80 images that needs replacing with higher res images since the originals was not kept in any sorted order.

So you kept the originals, but didn't organize them into a way you could use them later?

Re: How Discord Resizes 150M Images Every Day with Go and C++

#97

Post by Xeoncross »

drdaeman wrote: post #90
Xeoncross wrote: post #71

Earlier quoted context omitted.

Yes, the use-case of proxying images is a different mater. I was talking about client uploads since so many companies seem determined to waste my bandwidth and time uploading without resizing first.

Client uploads with client-provided thumbnails could lead to some pranks. You know, a thumbnail with a cute kitten but something completely different after you click on it ;)

I think you misunderstood. The client is uploading a single data blob for each image, but the blob can be smaller if the client resizes it first.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#98

Post by abeach222 »

kylehotchkiss wrote: post #43

I wish Cloudfront supported resize parameters so we wouldn't have to keep buildings these or paying a lot for Imgix.

You can use lambda edge functions for this. They recently announced support for query string parameters.

https://aws.amazon.com/about-aws/whats-new/2017/10/lambda-at...

I have built an image resizing service around this with go and libvips. With go libvips, s3gof3r, you can load s3 images directly into a buffer, pass to libvips, and serve without writing to disk. Basically, you can use edge functions with your origin as the above go service.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#99

Post by tzahola »

radicalbyte wrote: post #53
b1naryth1ef wrote: post #29

Earlier quoted context omitted.

Ah sorry I misunderstood you. We keep the original image around and provide different sizes for different platforms/resolutions/dpis/etc.

You should seriously consider doing this for your mobile client; the worst thing about Discord is that it eats mobile data if you're uploading lots of images.

That’s exactly the reason why the iOS Mail app asks you whether you want to attach an image in small/medium/large/original size.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#100

Post by hemancuso »

caltrops wrote: post #6

I’d be very worried about a security issue with the unsafe C++ code. You really have to run this kind of complex parsing in a disposable containerized environment to do it safely. Or do everything carefully and in a memory safe language.

I'd love to be pointed at any resource where somebody who has spent the time walks through the best way to do this safely. Is the only way to do it safely inside a container via some networked connection? Are there other ways to lock down ImageMagick etc such that you can resize safely?
Post reply on HN