Earlier quoted context omitted.
Our site would have been happy with full res images from the start. As it is now we are stuck with 80x80 images that needs replacing with higher res images since the originals was not kept in any sorted order.
So you kept the originals, but didn't organize them into a way you could use them later?
How Discord Resizes 150M Images Every Day with Go and C++
151–157 of 157 posts
Re: How Discord Resizes 150M Images Every Day with Go and C++
#152Earlier quoted context omitted.
Ah sorry I misunderstood you. We keep the original image around and provide different sizes for different platforms/resolutions/dpis/etc.
You should seriously consider doing this for your mobile client; the worst thing about Discord is that it eats mobile data if you're uploading lots of images.
Re: How Discord Resizes 150M Images Every Day with Go and C++
#153Earlier quoted context omitted.
Because then you don't get to build out fun infrastructure like this and write it up in your company blog.
Agreed. I wish the posts contained a "it cost X developer hours to recreate thumbor or $$$ total, and we saved Y dollars per month" meaning in approximately 15 years we'll have broken even on this investment. Oh yeah and we don't even do intelligent resizing like thumbor does.
Re: How Discord Resizes 150M Images Every Day with Go and C++
#154Earlier quoted context omitted.
Can you link to which resize library you're using? We'd love to see a 90% further reduction in instances
Sorry to be confusing, I am not resizing images. Just working with data sets as large as what I image 150M images would be. The software I am working on takes point and time backups of computers and uploads them to "the cloud", I mean servers in a data center. There they can be virtualized with a click of a button in mass or one at a time, and near instantly. This involves transfering, encrypting, compression and cre…
Re: How Discord Resizes 150M Images Every Day with Go and C++
#155Earlier quoted context omitted.
This isn't just resampling an image: decoding a variety of image (and even video) formats, decompressing the selected frame, performing the actual resize, and then compressing the result. If the resample doesn't save more than the setup overhead, it'd be an immediate loss. Even if it does, there's an engineering cost since you now need to make sure that all of your servers have GPUs available, your chosen implementat…
> This isn't just resampling an image GPUs can do that, too: http://fastcompression.com/products/jpeg/cuda-jpeg.htm > you now need to make sure that all of your servers have GPUs available OP is running on google’s cloud: “n1-standard-16 host type, peaking at 12 instances on a typical day.” That instance costs $0.76/hour. Adding NVIDIA Tesla K80 is $0.7 extra. > it's really tempting not to deal with any of that Yeah,…
> GPUs can do that, too: http://fastcompression.com/products/jpeg/cuda-jpeg.htm
Agreed - but for how many different formats, and how well do those implementations support all of the various format options for things like bit depth or palettes, compression variants, etc.? That's not just things like compliance testing – itself a big problem – but also handling all of the slightly non-compliant data in the wild which users will inevitably expect to work.
(I'm somewhat biased having spent time dealing with JPEG 2000 imagery where various lapses on the standards side meant that it's still common to find images which don't display correctly in one or more implementations but are silently reported as correct in others)
Again, I'm not arguing that doing this on a GPU isn't a good idea — the hardware has become common enough that it's reasonable to assume availability for anyone who cares — but just that there's significant overhead cost for anyone who needs to handle images from unconstrained sources. It'll happen but this kind of thing always takes longer than it seems like it should.
Re: How Discord Resizes 150M Images Every Day with Go and C++
#156Earlier quoted context omitted.
isn't it a meme at this point, pushing computational work to the client side? I have a laptop or mobile device, please don't hog my limited cpu and battery life by forcing my device to resize images.
Are you sure your wifi connection will not eat more power with huge upload than processor/gpu doing the scaling?
also -- there's also page load time. if it's an intensive calculation then the overhead of sending the results over http is still less than the browser calc time.
Re: How Discord Resizes 150M Images Every Day with Go and C++
#157Earlier quoted context omitted.
> This isn't just resampling an image GPUs can do that, too: http://fastcompression.com/products/jpeg/cuda-jpeg.htm > you now need to make sure that all of your servers have GPUs available OP is running on google’s cloud: “n1-standard-16 host type, peaking at 12 instances on a typical day.” That instance costs $0.76/hour. Adding NVIDIA Tesla K80 is $0.7 extra. > it's really tempting not to deal with any of that Yeah,…
> > This isn't just resampling an image > GPUs can do that, too: http://fastcompression.com/products/jpeg/cuda-jpeg.htm Agreed - but for how many different formats, and how well do those implementations support all of the various format options for things like bit depth or palettes, compression variants, etc.? That's not just things like compliance testing – itself a big problem – but also handling all of the slightl…
Flickr is doing just that, and they’ve been using GPUs for more than 2 years already:
http://code.flickr.net/2015/06/25/real-time-resizing-of-flic...
> It'll happen but this kind of thing always takes longer than it seems like it should.
I think the main reason for that is lazy software developers reluctant to learn new stuff.