Live data from Hacker News

How Discord Resizes 150M Images Every Day with Go and C++

blog.discordapp.com

151–157 of 157 posts

Re: How Discord Resizes 150M Images Every Day with Go and C++

#151
post #94

Earlier quoted context omitted.

Our site would have been happy with full res images from the start. As it is now we are stuck with 80x80 images that needs replacing with higher res images since the originals was not kept in any sorted order.

So you kept the originals, but didn't organize them into a way you could use them later?

Long story but from the start we kept originals organized. Then we restructured and the new people couldnt care less and threw away originals or left them named 1,2,3 and so on. All useless now.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#152

Earlier quoted context omitted.

Ah sorry I misunderstood you. We keep the original image around and provide different sizes for different platforms/resolutions/dpis/etc.

You should seriously consider doing this for your mobile client; the worst thing about Discord is that it eats mobile data if you're uploading lots of images.

The other side of this is if you're on mobile data and they decide to resize the images on the client then people will complain about the app eating up battery life because it's using so much CPU to resize the images. Plus, they won't have the full size image to share with people you may be chatting with on desktop. If you want to not use so much data while uploading images then you should probably resize them yourself or just not upload images unless you're on wifi.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#153
post #56
post #13

Earlier quoted context omitted.

Because then you don't get to build out fun infrastructure like this and write it up in your company blog.

Agreed. I wish the posts contained a "it cost X developer hours to recreate thumbor or $$$ total, and we saved Y dollars per month" meaning in approximately 15 years we'll have broken even on this investment. Oh yeah and we don't even do intelligent resizing like thumbor does.

It looks like thumbor is built on a Python stack similar to what Discord was using in their original service. What makes you say they didn't consider it or benchmark it against their previous service and make the decision that it wasn't as good as they needed?

Re: How Discord Resizes 150M Images Every Day with Go and C++

#154

Earlier quoted context omitted.

Can you link to which resize library you're using? We'd love to see a 90% further reduction in instances

Sorry to be confusing, I am not resizing images. Just working with data sets as large as what I image 150M images would be. The software I am working on takes point and time backups of computers and uploads them to "the cloud", I mean servers in a data center. There they can be virtualized with a click of a button in mass or one at a time, and near instantly. This involves transfering, encrypting, compression and cre…

Assuming the average image size is 3 MB which seems conservative, especially if they're handling GIFs as well, this is 450 TB per day. If you're handling that much data on one beefy machine then kudos.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#155
post #115

Earlier quoted context omitted.

This isn't just resampling an image: decoding a variety of image (and even video) formats, decompressing the selected frame, performing the actual resize, and then compressing the result. If the resample doesn't save more than the setup overhead, it'd be an immediate loss. Even if it does, there's an engineering cost since you now need to make sure that all of your servers have GPUs available, your chosen implementat…

> This isn't just resampling an image GPUs can do that, too: http://fastcompression.com/products/jpeg/cuda-jpeg.htm > you now need to make sure that all of your servers have GPUs available OP is running on google’s cloud: “n1-standard-16 host type, peaking at 12 instances on a typical day.” That instance costs $0.76/hour. Adding NVIDIA Tesla K80 is $0.7 extra. > it's really tempting not to deal with any of that Yeah,…

> > This isn't just resampling an image

> GPUs can do that, too: http://fastcompression.com/products/jpeg/cuda-jpeg.htm

Agreed - but for how many different formats, and how well do those implementations support all of the various format options for things like bit depth or palettes, compression variants, etc.? That's not just things like compliance testing – itself a big problem – but also handling all of the slightly non-compliant data in the wild which users will inevitably expect to work.

(I'm somewhat biased having spent time dealing with JPEG 2000 imagery where various lapses on the standards side meant that it's still common to find images which don't display correctly in one or more implementations but are silently reported as correct in others)

Again, I'm not arguing that doing this on a GPU isn't a good idea — the hardware has become common enough that it's reasonable to assume availability for anyone who cares — but just that there's significant overhead cost for anyone who needs to handle images from unconstrained sources. It'll happen but this kind of thing always takes longer than it seems like it should.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#156
post #148

Earlier quoted context omitted.

isn't it a meme at this point, pushing computational work to the client side? I have a laptop or mobile device, please don't hog my limited cpu and battery life by forcing my device to resize images.

Are you sure your wifi connection will not eat more power with huge upload than processor/gpu doing the scaling?

I dont see how that matters at all.

also -- there's also page load time. if it's an intensive calculation then the overhead of sending the results over http is still less than the browser calc time.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#157
post #155

Earlier quoted context omitted.

> This isn't just resampling an image GPUs can do that, too: http://fastcompression.com/products/jpeg/cuda-jpeg.htm > you now need to make sure that all of your servers have GPUs available OP is running on google’s cloud: “n1-standard-16 host type, peaking at 12 instances on a typical day.” That instance costs $0.76/hour. Adding NVIDIA Tesla K80 is $0.7 extra. > it's really tempting not to deal with any of that Yeah,…

> > This isn't just resampling an image > GPUs can do that, too: http://fastcompression.com/products/jpeg/cuda-jpeg.htm Agreed - but for how many different formats, and how well do those implementations support all of the various format options for things like bit depth or palettes, compression variants, etc.? That's not just things like compliance testing – itself a big problem – but also handling all of the slightl…

> significant overhead cost for anyone who needs to handle images from unconstrained sources

Flickr is doing just that, and they’ve been using GPUs for more than 2 years already:

http://code.flickr.net/2015/06/25/real-time-resizing-of-flic...

> It'll happen but this kind of thing always takes longer than it seems like it should.

I think the main reason for that is lazy software developers reluctant to learn new stuff.

Post reply on HN