Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Resizes 150M Images Every Day with Go and C++

blog.discordapp.com

Post reply on HN
Print view View original

121–130 of 157 posts

Re: How Discord Resizes 150M Images Every Day with Go and C++

#121

Post by megy »

codedokode wrote: post #113
Xeoncross wrote: post #8

Why don't more companies resize images client-side first using and then save the server some work by only asking it to verify the result by - resizing to the same size - removing metadata This results in much faster transfer (10x less bandwidth used often for mobile uploads) and reduces server load by "farming" out the work to the clients. https://developer.mozilla.org/en-US/docs/Web/API/CanvasRende... # Edit: On Kee…

As I understand they needed to download images from any external servers using an URL. I am not sure if that is possible even with CORS. Also they don't save preview images and generate them when needed as I understood. So what you are suggesting requires a lot of disk space to keep thumbnails that might be never needed later. And if you don't have millions of uploads per day then it makes no sense trying to save som…

I have no idea how what you are saying related to compressing images before they are sent?

Re: How Discord Resizes 150M Images Every Day with Go and C++

#122

Post by manigandham »

jhgg wrote: post #107
bpicolo wrote: post #105

Earlier quoted context omitted.

In this case, it was perhaps cheaper for them to do in-house, and it's not rocket science? They wrote a bleeding edge library for it - sounds like they have the expertise just fine. Minimizing external dependencies can be a big deal if you have the developers to manage it. Also, it is totally core to what they do. Images are a huge part of the Discord UX.

At 150m images per day, not counting bandwidth, imgix would cost ~135k/month. Running 12 n1-standard-16 instances (peak load according to the article) is ~$5k/month. It's not hard to see why we wrote it in house when you consider that cost.

Ok, so why a new library and associated dev time when thumbor and other libraries already exist, especially if you're willing to spend 5k/month on instances just for this?

Re: How Discord Resizes 150M Images Every Day with Go and C++

#123

Post by bpicolo »

manigandham wrote: post #122
jhgg wrote: post #107

Earlier quoted context omitted.

At 150m images per day, not counting bandwidth, imgix would cost ~135k/month. Running 12 n1-standard-16 instances (peak load according to the article) is ~$5k/month. It's not hard to see why we wrote it in house when you consider that cost.

Ok, so why a new library and associated dev time when thumbor and other libraries already exist, especially if you're willing to spend 5k/month on instances just for this?

That was pretty clear in the post - they didn't find a Golang lib that could compete with their pillow-simd on resizing, which was the main performance bottleneck.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#124

Post by gruez »

Operyl wrote: post #120
gruez wrote: post #109

Earlier quoted context omitted.

to be fair, you could say the same of facebook/google. they're not selling the data (giving it to third parties), but they're making money off it.

Sure, they serve said data to other authorized users and clients (intended recipients). Name a business that doesn’t do that. ;)

point is, their statement does not preclude them from using data for marketing purposes. some people are content with that, but others take it as a sign that they are (or will) use the data to build dossiers on users.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#125

Post by ymse »

This post reminded me of a very old article from Yahoo/Tumblr explaining how they were (ab)using Ceph to generate thumbnails on the fly as pictures were uploaded using the Ceph OSD plugin interface.

Unfortunately the post seems to have disappeared from the internet (it was probably around 6 years ago), so here are some other teasers:

https://yahooeng.tumblr.com/post/116391291701/yahoo-cloud-ob...

https://ceph.com/geen-categorie/dynamic-object-interfaces-wi...

Disclaimer: not affiliated with Ceph apart from being a happy sysadmin.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#126

Post by mbrumlow »

0xbear wrote: post #93

That’s 1700 images per second. Doable on one (beefy) box. 3 to account for the diurnal cycle. Am I supposed to be impressed?

I don't get why you are being down voted. This is almost exactly what I thought. It's just not that much data given the state is computer hardware.

Where I work we have single nodes processing near that much data a hour -- these are beefy systems though.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#127

Post by noahdesu »

ymse wrote: post #125

This post reminded me of a very old article from Yahoo/Tumblr explaining how they were (ab)using Ceph to generate thumbnails on the fly as pictures were uploaded using the Ceph OSD plugin interface. Unfortunately the post seems to have disappeared from the internet (it was probably around 6 years ago), so here are some other teasers: https://yahooeng.tumblr.com/post/116391291701/yahoo-cloud-ob... https://ceph.com/gee…

Here is a link to a talk I gave last month describing how to use Lua to generate thumbnails remotely in the Ceph/RADOS OSD servers.

Talk is from Lua workshop 2017. Relevant content begins at 15m40s.

https://youtu.be/bGQc-PpJAyk?t=15m40s

Re: How Discord Resizes 150M Images Every Day with Go and C++

#128

Post by manigandham »

bpicolo wrote: post #123
manigandham wrote: post #122

Earlier quoted context omitted.

Ok, so why a new library and associated dev time when thumbor and other libraries already exist, especially if you're willing to spend 5k/month on instances just for this?

That was pretty clear in the post - they didn't find a Golang lib that could compete with their pillow-simd on resizing, which was the main performance bottleneck.

Why was a Go version needed if performance was paramount? There are libraries already that can handle this performance just fine.

If they're going to spend 60k/year on instances, the dev time definitely wasn't worth it for this. They just wanted to use that language because this is a NIH situation, not really an engineering priority.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#129

Post by 0xbear »

mbrumlow wrote: post #126
0xbear wrote: post #93

That’s 1700 images per second. Doable on one (beefy) box. 3 to account for the diurnal cycle. Am I supposed to be impressed?

I don't get why you are being down voted. This is almost exactly what I thought. It's just not that much data given the state is computer hardware. Where I work we have single nodes processing near that much data a hour -- these are beefy systems though.

People have just drunk so much “cheap commodity hardware” kool aid by now, they don’t realize there are cheaper and easier ways of doing things now, assuming you have devs who can code and tune for performance. Same with “big data”. Most people have sub-1T datasets. You simply don’t need Spark or anything custom for that.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#130

Post by brian-armstrong »

0xbear wrote: post #93

That’s 1700 images per second. Doable on one (beefy) box. 3 to account for the diurnal cycle. Am I supposed to be impressed?

Can you link to which resize library you're using? We'd love to see a 90% further reduction in instances
Post reply on HN