Using C for a specialized data store
pixenomics.tumblr.com
Using C for a specialized data store
1–10 of 19 posts
Re: Using C for a specialized data store
#2Seems to me they skipped right over the most obvious option: Redis.
It's quite fast, you can use a Redis string as a random-access array up to 512MB/each, and there are several good ways to handle persistence/backup. I don't think there was a need for them to write any C themselves.
Re: Using C for a specialized data store
#3Seems to me they skipped right over the most obvious option: Redis. It's quite fast, you can use a Redis string as a random-access array up to 512MB/each, and there are several good ways to handle persistence/backup. I don't think there was a need for them to write any C themselves.
We were looking at that (as well as riak) but processing the data would require pulling all the data into PHP. I guess you could do the processing in C but it's then just as easy to store it there as well.
Re: Using C for a specialized data store
#4Seems to me they skipped right over the most obvious option: Redis. It's quite fast, you can use a Redis string as a random-access array up to 512MB/each, and there are several good ways to handle persistence/backup. I don't think there was a need for them to write any C themselves.
The 4th paragraph explained why they didn't go this way:
> We were reluctant to use a NoSQL solution as this would require retrieving the pixels through a socket, storing it in memory and then processing them. It makes more sense to process it where it’s stored.
Re: Using C for a specialized data store
#5Seems to me they skipped right over the most obvious option: Redis. It's quite fast, you can use a Redis string as a random-access array up to 512MB/each, and there are several good ways to handle persistence/backup. I don't think there was a need for them to write any C themselves.
We were looking at that (as well as riak) but processing the data would require pulling all the data into PHP. I guess you could do the processing in C but it's then just as easy to store it there as well.
Have you looked into the LUA scripting option for Redis? Allows for some processing to happen on the server side, and it's quite powerful.
Re: Using C for a specialized data store
#6Seems to me they skipped right over the most obvious option: Redis. It's quite fast, you can use a Redis string as a random-access array up to 512MB/each, and there are several good ways to handle persistence/backup. I don't think there was a need for them to write any C themselves.
We were looking at that (as well as riak) but processing the data would require pulling all the data into PHP. I guess you could do the processing in C but it's then just as easy to store it there as well.
I'm not clear why you're worried about that. Is it the pulling, or the processing?
The pulling shouldn't be an issue -- I don't know about PHP, but in pure Python, I can pull an arbitrary 10MB string from Redis in ~85-90ms. With hiredis (C extension), that falls to about 47ms.
I can't speak to processing, since I don't know exactly what transformations you're performing.
Re: Using C for a specialized data store
#7Earlier quoted context omitted.
We were looking at that (as well as riak) but processing the data would require pulling all the data into PHP. I guess you could do the processing in C but it's then just as easy to store it there as well.
I'm not clear why you're worried about that. Is it the pulling, or the processing? The pulling shouldn't be an issue -- I don't know about PHP, but in pure Python, I can pull an arbitrary 10MB string from Redis in ~85-90ms. With hiredis (C extension), that falls to about 47ms. I can't speak to processing, since I don't know exactly what transformations you're performing.
It's more the iteration of each pixel and it's neighbor (of which there are 8) making it around 9.6 million iterations.
We will probably head towards redis in the future when precise backups are essential. Undecided what will do this processing though.
Re: Using C for a specialized data store
#8Earlier quoted context omitted.
We were looking at that (as well as riak) but processing the data would require pulling all the data into PHP. I guess you could do the processing in C but it's then just as easy to store it there as well.
Have you looked into the LUA scripting option for Redis? Allows for some processing to happen on the server side, and it's quite powerful.
That sounds like a good option. Thanks, will note it.
Re: Using C for a specialized data store
#9Umm, we're talking about just over 18MB here (1200 * 1000 pixels, 16 bytes/pixel, see http://pixenomics.tumblr.com/post/16895861678/how-to-send-1-...). That you can just dump over the wire as a binary blob. Why are we talking about this again? Use your favourite language, just keep it in a big blob in memory, and have fun.
Re: Using C for a specialized data store
#10Earlier quoted context omitted.
I'm not clear why you're worried about that. Is it the pulling, or the processing? The pulling shouldn't be an issue -- I don't know about PHP, but in pure Python, I can pull an arbitrary 10MB string from Redis in ~85-90ms. With hiredis (C extension), that falls to about 47ms. I can't speak to processing, since I don't know exactly what transformations you're performing.
It's more the iteration of each pixel and it's neighbor (of which there are 8) making it around 9.6 million iterations. We will probably head towards redis in the future when precise backups are essential. Undecided what will do this processing though.
We built GPU-accelerated NoSQL datastore. using it, this can be accelerated 100x, given you switch to binary pixel format.