Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Resizes 150M Images Every Day with Go and C++

blog.discordapp.com

Post reply on HN
Print view View original

81–90 of 157 posts

Re: How Discord Resizes 150M Images Every Day with Go and C++

#81

Post by Franciscouzo »

abiox wrote: post #52
bri3d wrote: post #15

Earlier quoted context omitted.

I'm not sure why this is being downvoted - image processing is one of the most dangerous parts of a common consumer-facing web software stack. By and large this is because image container formats are poorly documented, overly broad, and rely on a lot of tricky binary parsing that's easy to mess up in an unsafe programming language. It's also one of the most obvious ingress points for untrusted binary data uploaded by…

> I'm not sure why this is being downvoted if i was a betting person, i'd wager that it may see somewhat like "rewrite it in rust" cargo culting.

Except those would be upvoting it...

Re: How Discord Resizes 150M Images Every Day with Go and C++

#82

Post by fleitz »

caltrops wrote: post #67
Impossible wrote: post #55

Earlier quoted context omitted.

The downvotes are also because it's a somewhat cliche comment on HN now. Anytime anyone is doing any with C or C++ that is even indirectly web facing, "this could be unsafe!!!" is an obligatory comment, even though all major tech companies have core components written in C++, and there are big web apps that have been running for years that are mostly written in C or C++. Security is definitely a concern, but these ki…

This isn’t one of those. Handing large amounts of unvalidated user input to these libraries is particularly dangerous.

To be fair most everything under the hood passes through to these libraries. So even sticking with python means passing unvalidated blobs through to libpng/jpeg/tiff or some other low level language.

It's the entire reason python is generally fast enough, anything that's slow generally uses a C lib under the hood anyway.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#83

Post by manigandham »

zitterbewegung wrote: post #76
manigandham wrote: post #74

Nice, but why? https://cloudinary.com , https://www.imgix.com , or https://www.filestack.com already exist and are well worth it for 99% of apps. Even at scale, it really doesn't cost that much to have someone else do it. You can use a thin proxy through your existing CDN if you want to save on their bandwidth fees. Also http://thumbor.org and https://imageresizing.net if you want a library to host yourself which are…

Maybe it’s because that they don’t want a dependency on a external service that could go down ?

It's images... seems like a very low risk situation, especially when they are served from a CDN.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#84

Post by manigandham »

kylehotchkiss wrote: post #77
fleitz wrote: post #59

Earlier quoted context omitted.

How much would you pay for an image resizing service? I'd been thinking for a while of putting a fleet of autoscaled thumbor boxes behind cloudfront and making a billing API for it.

Imgix's $10 minimum is so much for a personal site with maybe 500 uniques a month. If you're going for a service like that, think of people like me who host on s3/cloudfront for $.20/month. But let people scale up to millions of pageviews a month. Don't need anything fancy. Just w=? h=? would be great, developers can handle the DPI stuff with sourceset tags.

Cloudinary is free. https://cloudinary.com/pricing

Re: How Discord Resizes 150M Images Every Day with Go and C++

#85

Post by brightball »

fleitz wrote: post #68
Karrot_Kream wrote: post #66

Earlier quoted context omitted.

I think that's a bit reductionist, no? There are many reasons they may have been searching for moving to Go. Off the top of my head I can think of: 1. Static typing increasing confidence and velocity 2. Better developer-facing tooling increasing velocity 3. More employees knowledgeable about Go than Python 4. More enthusiasm (and therefore faster velocity) around Go development. The blog post was about the engineerin…

It might be, then again I see a lot of wheel reinvention in tech / NIH syndrome. I'm the kind of hacker who if a service runs out of memory every 2 hours, writes a crontab to restart it every hour after X random minutes so they don't all restart at the same time. It gets a lot of eye rolls from the other engineers searching for perfection, but it tends to produce services quickly that are highly reliable. And look no…

Part of it is just Discord’s operating scale. They are already leveraging Elixir clustering to an extremely high rate of concurrency and when you start thinking about problems from that standpoint Go becomes a much more natural fit within the stack for low level micro services.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#87

Post by reificator »

manigandham wrote: post #83
zitterbewegung wrote: post #76

Earlier quoted context omitted.

Maybe it’s because that they don’t want a dependency on a external service that could go down ?

It's images... seems like a very low risk situation, especially when they are served from a CDN.

As a user, the order of importance for Discord services is:

* Voice

* Text

* Previews (images, gifs, and videos)

Previews going down would be a pretty big deal for my communities based on the way we use the platform.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#88

Post by user5994461 »

Xeoncross wrote: post #8

Why don't more companies resize images client-side first using and then save the server some work by only asking it to verify the result by - resizing to the same size - removing metadata This results in much faster transfer (10x less bandwidth used often for mobile uploads) and reduces server load by "farming" out the work to the clients. https://developer.mozilla.org/en-US/docs/Web/API/CanvasRende... # Edit: On Kee…

Historically, resizing images client side doesn't work because most clients are not able to render the images, let alone resize them.

The image file formats are very very complicated, many are platform specific, some are covered by patents.

For example of a common issue, another comment mentioned the rotation parameter, it's set by many cameras but the support is inconsistent.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#89

Post by Volt »

StreamBright wrote: post #80
zitterbewegung wrote: post #76

Earlier quoted context omitted.

Maybe it’s because that they don’t want a dependency on a external service that could go down ?

So you have an internal dependency that could go down?

It's better when you can do something about it.

Re: How Discord Resizes 150M Images Every Day with Go and C++

#90

Post by drdaeman »

Xeoncross wrote: post #71
Lutin wrote: post #49

Earlier quoted context omitted.

This is for proxying images that users link in chat, not for when users upload images to the service. It doesn't make sense to talk about doing this resize on the client, as the client doesn't have the image.

Yes, the use-case of proxying images is a different mater. I was talking about client uploads since so many companies seem determined to waste my bandwidth and time uploading without resizing first.

Client uploads with client-provided thumbnails could lead to some pranks.

You know, a thumbnail with a cute kitten but something completely different after you click on it ;)

Post reply on HN