How Discord Resizes 150M Images Every Day with Go and C++
blog.discordapp.com
How Discord Resizes 150M Images Every Day with Go and C++
1–10 of 157 posts
Re: How Discord Resizes 150M Images Every Day with Go and C++
#2Did it seem to anyone else that sticking to Python would have been way easier? It didn’t seem like any of the performance gains were through Golang.
Re: How Discord Resizes 150M Images Every Day with Go and C++
#3Link to the resulting open-source project:
Re: How Discord Resizes 150M Images Every Day with Go and C++
#4Did it seem to anyone else that sticking to Python would have been way easier? It didn’t seem like any of the performance gains were through Golang.
Yeah, either a good Python JIT or Cython would have been fine honestly. I never understood the obsession with "python is slow" when you can recover almost all of the performance with a good JIT or Cython (in many/most cases).
Re: How Discord Resizes 150M Images Every Day with Go and C++
#5Did it seem to anyone else that sticking to Python would have been way easier? It didn’t seem like any of the performance gains were through Golang.
Yeah, either a good Python JIT or Cython would have been fine honestly. I never understood the obsession with "python is slow" when you can recover almost all of the performance with a good JIT or Cython (in many/most cases).
What are some good Python JIT's that are worth trying out?
Re: How Discord Resizes 150M Images Every Day with Go and C++
#6I’d be very worried about a security issue with the unsafe C++ code.
You really have to run this kind of complex parsing in a disposable containerized environment to do it safely. Or do everything carefully and in a memory safe language.
Re: How Discord Resizes 150M Images Every Day with Go and C++
#7Anybody knows how well libvips https://github.com/DAddYE/vips compares to liliput performance wise?
Re: How Discord Resizes 150M Images Every Day with Go and C++
#8Why don't more companies resize images client-side first using and then save the server some work by only asking it to verify the result by
- resizing to the same size
- removing metadata
This results in much faster transfer (10x less bandwidth used often for mobile uploads) and reduces server load by "farming" out the work to the clients.
https://developer.mozilla.org/en-US/docs/Web/API/CanvasRende...
# Edit: On Keeping Full Resolution Images
Some people mention having original highest-resolution images are important. I don't think that is true for most applications.
Most apps don't need hi-resolution history as much as current, live engagement so older photos being smaller isn't a big deal. As technology moves on you simply start allowing higher-res uploads. Youtube, facebook, and others have done this fine as the older stuff is replaced with the new/current/now() content.
In fact, even our highest resolution images are still low-quality for the future. Pick a good max size for your site (4k?) and resize everything down to that. In a year, bump it up to 6k, then 10k, etc...
Keeping costs low has it's benefits, especially for us startups. Now if you have massive collateral, then knock yourself out.
Re: How Discord Resizes 150M Images Every Day with Go and C++
#9Did it seem to anyone else that sticking to Python would have been way easier? It didn’t seem like any of the performance gains were through Golang.
From reading this, seems HTTP handling speed was important to them? which Go is probably better for. Also, interfacing Python to C/C++ is pretty unpleasant.
Re: How Discord Resizes 150M Images Every Day with Go and C++
#10Did it seem to anyone else that sticking to Python would have been way easier? It didn’t seem like any of the performance gains were through Golang.
I believe this little piece answers your question:
> We likely could have addressed this behavior in Image Proxy, but we had been experimenting with using more Go, and it seemed like a good place to try Go out.
At the heart of if, they were looking for opportunities to use more Go in their stack and they deemed this situation as a fit.