Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Resizes 150M Images Every Day with Go and C++

blog.discordapp.com

Post reply on HN
Print view View original

131–140 of 157 posts

Re: How Discord Resizes 150M Images Every Day with Go and C++

#131

Post by brian-armstrong »

manigandham wrote: post #128
bpicolo wrote: post #123

Earlier quoted context omitted.

That was pretty clear in the post - they didn't find a Golang lib that could compete with their pillow-simd on resizing, which was the main performance bottleneck.

Why was a Go version needed if performance was paramount? There are libraries already that can handle this performance just fine. If they're going to spend 60k/year on instances, the dev time definitely wasn't worth it for this. They just wanted to use that language because this is a NIH situation, not really an engineering priority.

We specifically addressed this question in our post. Not only did we reduce from ~25 instances to ~12, we also added new features.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#132

Post by ehsankia »

radicalbyte wrote: post #53
b1naryth1ef wrote: post #29

Earlier quoted context omitted.

Ah sorry I misunderstood you. We keep the original image around and provide different sizes for different platforms/resolutions/dpis/etc.

You should seriously consider doing this for your mobile client; the worst thing about Discord is that it eats mobile data if you're uploading lots of images.

Data aside, my phone takes pictures at ridiculously high resolutions. Whenever I go to send pictures with discord, it takes a good 30 seconds and half the time it'll just break.

As an aside, I wish the "share" button would share a lower resolution image instead. I don't mind storing the full quality picture, but handling a 10mb image is seriously silly.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#134

Post by dajohnson89 »

Xeoncross wrote: post #8

Why don't more companies resize images client-side first using and then save the server some work by only asking it to verify the result by - resizing to the same size - removing metadata This results in much faster transfer (10x less bandwidth used often for mobile uploads) and reduces server load by "farming" out the work to the clients. https://developer.mozilla.org/en-US/docs/Web/API/CanvasRende... # Edit: On Kee…

isn't it a meme at this point, pushing computational work to the client side? I have a laptop or mobile device, please don't hog my limited cpu and battery life by forcing my device to resize images.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#135

Post by searealist »

malux85 wrote: post #61
AndrewStephens wrote: post #16

Earlier quoted context omitted.

A few reasons: 1) Although the site serves up images at 1024 pixels (or whatever) today, in the future they may want larger images. When everyone is rocking 10K monitors and 6K phone displays, those small images are going to look pretty bad. 2) The original image has some metadata that they want to keep (geolocation, etc). 3) They think they can do a better and more consistent job resizing than the various browsers,…

agree on 3) most browsers just use linear interpolation when resizing images, which makes sense from a performance point of view, but looks terrible. Better to use a bi-linear or cubic resize, more computing up front, but better images, this is probably the reason they do it

bi-linear is linear interpolation

Re: How Discord Resizes 150M Images Every Day with Go and C++

#136

Post by searealist »

caltrops wrote: post #6

I’d be very worried about a security issue with the unsafe C++ code. You really have to run this kind of complex parsing in a disposable containerized environment to do it safely. Or do everything carefully and in a memory safe language.

This has nothing to do with parsing.

Also, your life must be very stressful.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#137

Post by [deleted] »

caltrops wrote: post #67
Impossible wrote: post #55

Earlier quoted context omitted.

The downvotes are also because it's a somewhat cliche comment on HN now. Anytime anyone is doing any with C or C++ that is even indirectly web facing, "this could be unsafe!!!" is an obligatory comment, even though all major tech companies have core components written in C++, and there are big web apps that have been running for years that are mostly written in C or C++. Security is definitely a concern, but these ki…

This isn’t one of those. Handing large amounts of unvalidated user input to these libraries is particularly dangerous.

[deleted]

Re: How Discord Resizes 150M Images Every Day with Go and C++

#138

Post by searealist »

caltrops wrote: post #67
Impossible wrote: post #55

Earlier quoted context omitted.

The downvotes are also because it's a somewhat cliche comment on HN now. Anytime anyone is doing any with C or C++ that is even indirectly web facing, "this could be unsafe!!!" is an obligatory comment, even though all major tech companies have core components written in C++, and there are big web apps that have been running for years that are mostly written in C or C++. Security is definitely a concern, but these ki…

This isn’t one of those. Handing large amounts of unvalidated user input to these libraries is particularly dangerous.

Unvalidated user input? What are you talking about, this is about image resizing. Your buzzwords make no sense.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#139

Post by LambdaComplex »

searealist wrote: post #138
caltrops wrote: post #67

Earlier quoted context omitted.

This isn’t one of those. Handing large amounts of unvalidated user input to these libraries is particularly dangerous.

Unvalidated user input? What are you talking about, this is about image resizing. Your buzzwords make no sense.

Yes, and images are user input in this case

Re: How Discord Resizes 150M Images Every Day with Go and C++

#140

Post by mbrumlow »

brian-armstrong wrote: post #130
0xbear wrote: post #93

That’s 1700 images per second. Doable on one (beefy) box. 3 to account for the diurnal cycle. Am I supposed to be impressed?

Can you link to which resize library you're using? We'd love to see a 90% further reduction in instances

Sorry to be confusing, I am not resizing images. Just working with data sets as large as what I image 150M images would be. The software I am working on takes point and time backups of computers and uploads them to "the cloud", I mean servers in a data center. There they can be virtualized with a click of a button in mass or one at a time, and near instantly.

This involves transfering, encrypting, compression and creating checksum of terabytes of data a hour (per node). While not exactly resizing images, I would image the computational power was on par with the service described. The entire system has about 4 PB or 8 PB in it right now, as backups are pruned (based on what people will pay for storage).

My software has a ton of space to grow and become better, but I think a better story would have been how discord handles 150M images a hour. If anything bandwidth acquiring the source image would be what I would consider the largest problem, not the CPU time to resize. In fact as long as your resize code slightly faster than the download then streaming it in and out would put your bottleneck entirely on bandwidth.

I will also note I am not a fan of libraries :p but that is not what this is about.

EDIT:

Also kudos to you, somebody criticized your post and you had the best response one could have. Inquiring minds are awesome.

Post reply on HN