Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ex-YouTube Man Builds Graphics Card for Entire Internet

wired.com

Post reply on HN
Print view View original

31–36 of 36 posts

Re: Ex-YouTube Man Builds Graphics Card for Entire Internet

#31

Post by benjamincburns »

300bps wrote: post #28
pavs wrote: post #10

Is this really such a hard problem to solve? Something like littleutils can easily solve this, no? http://littleutils.sourceforge.net/ Also what does this has to do with being "graphics card for the entire internet"?

>Is this really such a hard problem to solve? I must be misunderstanding because resizing an image, rotating it or many other basic operations consists of about 5 lines of .NET code and very little processing power. I imagine it's not any more difficult in other frameworks. I couldn't conceive of offloading that to an external service under just about any use case. It seems like they plan to offer video editing via S…

> 5 lines of .NET code

Yep.

> very little processing power

Not really. Those 5 lines are hiding a pretty big chunk of smarts. Simple scale/rotate transformations are an O(n) operation. That's not so bad. However image encoding/decoding is a bit more expensive.

JPEG encoding, for instance, utilizes an O(n*log(n)) frequency analysis algorithm called a Discrete Cosine Transform. Good JPEG encoders will also do some expensive analysis to maximize precision while minimizing size during quantization (edit: though most just use a set of standard quantization matrices). And then there's I/O. It seems fast when you're only processing one image at a time, but to do this as a real-time service for a number of high-traffic websites is very nontrivial.

Re: Ex-YouTube Man Builds Graphics Card for Entire Internet

#32

Post by jtanner »

cunac wrote: post #26
jtanner wrote: post #16

I host large images between 500k and 700k. I store the original image along with 4 different resized versions. The resized images on average account for 35% of disk space. I store about 300GBs of images. The resized versions take up about 100GBs of the 300. If I were to then to add retina versions (2x) of each thumbnail, I might be running at 70%+ disk space being used up by thumbnails. imgix is very interesting, as…

how 300GB is storage problem? I would understand if you say 300TB , but with 2TB HDD < 100$ and 0.5TB SSD < 400$ I can't really see that a major use case. And that is one time fixed payment.

Agreed. Unfortunately this site has many images and low traffic ( non-profit ), so it doesn't justify the cost of dedicated hardware.

Re: Ex-YouTube Man Builds Graphics Card for Entire Internet

#33

Post by hcarvalhoalves »

I was working on similar service based on previous project of mine (https://github.com/hcarvalhoalves/django-rest-thumbnails). I thought it was novel.

But well, Ex-YouTube? They have way more money to throw at this. I'll have to put that idea aside.

Re: Ex-YouTube Man Builds Graphics Card for Entire Internet

#35

Post by astrange »

benjamincburns wrote: post #31
300bps wrote: post #28

Earlier quoted context omitted.

>Is this really such a hard problem to solve? I must be misunderstanding because resizing an image, rotating it or many other basic operations consists of about 5 lines of .NET code and very little processing power. I imagine it's not any more difficult in other frameworks. I couldn't conceive of offloading that to an external service under just about any use case. It seems like they plan to offer video editing via S…

> 5 lines of .NET code Yep. > very little processing power Not really. Those 5 lines are hiding a pretty big chunk of smarts. Simple scale/rotate transformations are an O(n) operation. That's not so bad. However image encoding/decoding is a bit more expensive. JPEG encoding, for instance, utilizes an O(n*log(n)) frequency analysis algorithm called a Discrete Cosine Transform. Good JPEG encoders will also do some expe…

> JPEG encoding, for instance, utilizes an O(n*log(n)) frequency analysis algorithm called a Discrete Cosine Transform.

Scuse me?

JPEG's frequency transform is a set of fixed linear transforms and is always 8x8, so this step is just O(n) of the area of the image. Also, it's not really "a DCT", just any approximation that will look good when run through the approximation of the opposite transform at the other side.

See http://multimedia.cx/eggs/dct-pr/.

The compression steps after are the most difficult, since they have poor ILP and can involve division. But JPEG is really simple, it's not a burden these days.

One interesting part of it is minimizing the memory use of your application, by using the smallest possible ring buffer for pixels rather than keeping the whole image in memory at once.

Re: Ex-YouTube Man Builds Graphics Card for Entire Internet

#36

Post by mhmtmhmt »

audionerd wrote: post #22

If you're considering the DIY server-side image resize route, here's a good list of options: https://github.com/adamdbradley/foresight.js/wiki/Server-Resizing-Images ... and a comparison of benchmarks – ImageMagick vs. competitors like the very fast, low memory VIPS: http://www.vips.ecs.soton.ac.uk/index.php?title=Speed_and_Memory_Use

VIPS values are very good and you should notice it says "All timings are for a 5,000 by 5,000 pixel 8-bit RGB image in uncompressed tiled TIFF format, 128 by 128 pixel tiles". I haven't used VIPS in production. It seems like it's better for big files. I heard VIPS is not that performing with small files.
Post reply on HN