Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lorem Picsum, death by a million pixel-gigabits

dmarby.se

Post reply on HN
Print view View original

21–30 of 55 posts

Re: Lorem Picsum, death by a million pixel-gigabits

#21

Post by N_A_T_E »

boyter wrote: post #16

Image resizer scaling is one of the more interesting problems I have worked on in the last 10 years of so. I was part of a small team that designed and built the resizer that powers the nine.com.au network of sites. Modest by USA standards it gets close to hundreds of millions of views a day across the whole network. We ended up using shared nothing architecture. The whole thing ran on 6 T2 large AWS instances using…

Six t2.large instances sounds pretty efficient. For high volume image resizing on mobile devices we have access to GPU libraries. I wonder if something CUDA or openCL powered would help increase efficiency in a cloud based service.

Re: Lorem Picsum, death by a million pixel-gigabits

#22

Post by haasted »

kingkool68 wrote: post #17
elcomet wrote: post #2

I wonder how much it costs them to maintain this service.

I run https://dummyimage.com , the first placeholder image service which has been online since 2007. I use a 1GB, 1vCPU instance from DreamCompute costing $6.00 per month. That's it. To be fair dummyimage.com doesn't do any sort of image reading or resizing. Source doe at https://github.com/kingkool68/dummyimage

What kind of traffic does your service attract?

Re: Lorem Picsum, death by a million pixel-gigabits

#23

Post by tiborsaas »

AstralStorm wrote: post #19
tiborsaas wrote: post #15

Earlier quoted context omitted.

It's a good exercise to test these technologies and how to run things at scale. A real image from third party is valuable since gradients don't have onload events for example.

An image gradient then. 1 px wide perhaps, or high. Few problems resizing it (even on client) and it should be reusable.

I think the point is not to get away with the scaling problem.

Here's one service I used in a project's development mode:

https://dummyimage.com/2500x2000/000/09f.png&text=HackerNews

In the production build I changed it to the real host.

Re: Lorem Picsum, death by a million pixel-gigabits

#24

Post by cabaalis »

swebs wrote: post #20
k2xl wrote: post #8

I wonder how much it would cost to do a similar service and just upload all images and dimension possibilities to S3 + use cloudfront cdn.

Well let's limit the possibilities to any image between 1x1 and 1920x1080. Calculating the number of possible images is simple enough. It's just 1920*1080, or 2073600 images. Now what's the average size of each image? Well the average of each dimension is half of the full size, so the average area should be 1/4th to of a 1920x1080 image, or 518400 pixels per image. So in total, we need to save 1074954240000 pixels. N…

I'll bet that most image requests will be within certain parameters. 2^x by 2^y. So you could probably pre-cache most real-world image sizes, and leave dynamic generation for 1-offs.

Re: Lorem Picsum, death by a million pixel-gigabits

#25

Post by lonelappde »

swebs wrote: post #20
k2xl wrote: post #8

I wonder how much it would cost to do a similar service and just upload all images and dimension possibilities to S3 + use cloudfront cdn.

Well let's limit the possibilities to any image between 1x1 and 1920x1080. Calculating the number of possible images is simple enough. It's just 1920*1080, or 2073600 images. Now what's the average size of each image? Well the average of each dimension is half of the full size, so the average area should be 1/4th to of a 1920x1080 image, or 518400 pixels per image. So in total, we need to save 1074954240000 pixels. N…

You can easily cut that by a factor of 100 simply by serving only multiples of 10pixels for sizes over 300px.

And cut by another factor of 10 for jpg compression

Re: Lorem Picsum, death by a million pixel-gigabits

#26

Post by baybal2 »

boyter wrote: post #16

Image resizer scaling is one of the more interesting problems I have worked on in the last 10 years of so. I was part of a small team that designed and built the resizer that powers the nine.com.au network of sites. Modest by USA standards it gets close to hundreds of millions of views a day across the whole network. We ended up using shared nothing architecture. The whole thing ran on 6 T2 large AWS instances using…

I once tried to pitch a one "mobile ecommerce website as a service" company in Vancouver to go for GPU based image rescaler at around 2011.

A very dumb proposal: no caching, resize on the fly, the gpu has many gigabits of resizing performance for as long as JPEG is involved. One GPU works in decoding with VDPAU, one in encoding with CUDA.

That knocked down any google app engine based "elastic" service on economic basis, but the catch is that you have to send that GPU resizer to every colo. That did not work out as with google app engine you were getting access for google's POP network for almost free, and they were already paying for gigantic amount of CDN traffic.

----

When I worked as sub-subcontractor for the Alibaba's RDMA wired DC project, there was one demo by another team where they got DSP devs involved and they got a 10GB/s JPEG transcoders for under 100W. I think, most of power budget was going to the FPGA that was linking it all with the NIC :/

An expensive toy, but it again demonstrated to me just how powerful is the "lockdown" power of all those "cloud" companies. You can not buy anything like this on the open market.

Imagine what it could've been if they offered something more cash worthy over the RDMA there.

I said long ago that the killed product during the Bitcoin boom was not the mining itself, but leasing and renting the rigs. Your capital costs get covered near instantly, and you can cash out the next week. I believe that all that "cloud" thing will eventually follow this path.

Re: Lorem Picsum, death by a million pixel-gigabits

#27

Post by bwilliams18 »

overcast wrote: post #18
k2xl wrote: post #8

I wonder how much it would cost to do a similar service and just upload all images and dimension possibilities to S3 + use cloudfront cdn.

Or just use Amazons serverless image handler, to server up requested sizes, and cache them. https://aws.amazon.com/solutions/serverless-image-handler/

This is what I use in production right now. It works well once you figure it out, but backwards compatibility with the previous Thumbor based version isn't as good (or at least correctly documented) as they say. And the previous Thumbor based processor just stopped working one day. It's not as robust as a system as I expected.

Re: Lorem Picsum, death by a million pixel-gigabits

#28

Post by overcast »

bwilliams18 wrote: post #27
overcast wrote: post #18

Earlier quoted context omitted.

Or just use Amazons serverless image handler, to server up requested sizes, and cache them. https://aws.amazon.com/solutions/serverless-image-handler/

This is what I use in production right now. It works well once you figure it out, but backwards compatibility with the previous Thumbor based version isn't as good (or at least correctly documented) as they say. And the previous Thumbor based processor just stopped working one day. It's not as robust as a system as I expected.

That's unfortunate, I've only just begun rolling into a new project, so I can't comment on how it used to be.

Re: Lorem Picsum, death by a million pixel-gigabits

#29

Post by emilfihlman »

kenforthewin wrote: post #5
elcomet wrote: post #2

I wonder how much it costs them to maintain this service.

The article mentions Digital Ocean provides the infrastructure so it seems the only cost is dev time. It reads almost like a promotional for DO with all of their services mentioned, which is probably why they support it to begin with.

I though about that, too, but that's not any different from using AWS specific services. I actually now realized why they use bloody annoying different names: it's free marketing when someone writes about how they implemented stuff instead of having a generic name.

Re: Lorem Picsum, death by a million pixel-gigabits

#30

Post by emilfihlman »

swebs wrote: post #20
k2xl wrote: post #8

I wonder how much it would cost to do a similar service and just upload all images and dimension possibilities to S3 + use cloudfront cdn.

Well let's limit the possibilities to any image between 1x1 and 1920x1080. Calculating the number of possible images is simple enough. It's just 1920*1080, or 2073600 images. Now what's the average size of each image? Well the average of each dimension is half of the full size, so the average area should be 1/4th to of a 1920x1080 image, or 518400 pixels per image. So in total, we need to save 1074954240000 pixels. N…

It's also possible to have the same base image that you cut a portion out of, so you wouldn't need to store an original image for each possibility, just cache results of cropping as needed.
Post reply on HN