11–19 of 19 posts
Earlier quoted context omitted.
It's more the iteration of each pixel and it's neighbor (of which there are 8) making it around 9.6 million iterations. We will probably head towards redis in the future when precise backups are essential. Undecided what will do this processing though.
We built GPU-accelerated NoSQL datastore. using it, this can be accelerated 100x, given you switch to binary pixel format.
Umm, we're talking about just over 18MB here (1200 * 1000 pixels, 16 bytes/pixel, see http://pixenomics.tumblr.com/post/16895861678/how-to-send-1-... ). That you can just dump over the wire as a binary blob. Why are we talking about this again? Use your favourite language, just keep it in a big blob in memory, and have fun.
Umm, we're talking about just over 18MB here (1200 * 1000 pixels, 16 bytes/pixel, see http://pixenomics.tumblr.com/post/16895861678/how-to-send-1-... ). That you can just dump over the wire as a binary blob. Why are we talking about this again? Use your favourite language, just keep it in a big blob in memory, and have fun.
Memory isn't an issue. It's processing the data and turning the storage into a format the client can read. A big blob isn't easy to send to the client unless it's an image or something and then it becomes an issue when you want to manipulate the data or process it.
Seems to me they skipped right over the most obvious option: Redis. It's quite fast, you can use a Redis string as a random-access array up to 512MB/each, and there are several good ways to handle persistence/backup. I don't think there was a need for them to write any C themselves.
The 4th paragraph explained why they didn't go this way: > We were reluctant to use a NoSQL solution as this would require retrieving the pixels through a socket, storing it in memory and then processing them. It makes more sense to process it where it’s stored.
Earlier quoted context omitted.
Memory isn't an issue. It's processing the data and turning the storage into a format the client can read. A big blob isn't easy to send to the client unless it's an image or something and then it becomes an issue when you want to manipulate the data or process it.
it's a 1000x1000 image... this is a trivial problem
Earlier quoted context omitted.
The 4th paragraph explained why they didn't go this way: > We were reluctant to use a NoSQL solution as this would require retrieving the pixels through a socket, storing it in memory and then processing them. It makes more sense to process it where it’s stored.
Maybe I don't understand the problem,but that sounds like some serious premature optimization. 1.2 mpix is not much data.
Earlier quoted context omitted.
Maybe I don't understand the problem,but that sounds like some serious premature optimization. 1.2 mpix is not much data.
According to the article their Node solution took 4 seconds to run (down from 7 seconds after some optimization) and their C solution 0.03 seconds. Now maybe they could have sped up their node code more, but those sort of improvements hardly count as premature optimization.
Earlier quoted context omitted.
We built GPU-accelerated NoSQL datastore. using it, this can be accelerated 100x, given you switch to binary pixel format.
Why would you use a GPU-accelerated storage when latency is the main goal?
Use APU / HPU, if PCIe latency is a problem.
I understood that they running something like convolution (I.e, each pixel calculated from surrounding pixels) - this will be fast using OpenCL model).