I wish the tags page actually included frequency counts. And coud be sorted by frequency.
CKAN - The CPAN of data
11–15 of 15 posts
Re: CKAN - The CPAN of data
#12Every time something interesting that I want to read shows up on the front page, and clicking on it yields a spinning tab while the other end flaps about in agony, I can't help but wonder how many administrators know how to use ab(8). The rate of dead sites on HN is really a shocker, since commodity cloud that can stand up to HN's load -- as opposed to the shared hosting of yesterdecade -- is so widely available. I w…
Hi, I'm one of the CKAN devs. Just wanted to say the site is fully functional again (we've up the cached to be a bit more aggressive). As a side note we have indeed tested with ab :) Our problem is we continue to find the AWS instances we use somewhat unpredictable in their response to load (largely due, we believe, to the fluctuations in CPU "stealing" as load varies across the other instances that share the same ph…
Have you considered implementing some sort of script to scan some of the large biological databases and add links/metadata for the datasets they contain?
Looking at what's in CKAN now, it seems that it's mostly datasets that are a bit more easily understood than most of the biological data that's out there, but at the same time indexing and accessing biological data is a HUGE problem for researchers in this field.
There are currently some big databases such as the data stored by the UCSC genome browser (genome.ucsc.edu/downloads.html) and all sorts of expression/small RNA data available from GEO (Gene Expression Omnibus, http://www.ncbi.nlm.nih.gov/geo/), and lots of other slightly more esoteric databases like flybase.org, which specializes in fruit fly data.
Truly doing a proper job of indexing/classifying all of this is a close-to-impossible task (and in many cases requires specialized knowledge), but there are an absurd number of publicly available biological datasets out there. If you wanted to rapidly expand the number of entries you have you could use a script to index one or two of the big databases like GEO, and fill in the metadata from what they already have.
Of course, I can also understand why you might prefer to have the majority of the datasets in your site be the sort of thing most people (or at least, non-biologists) can interpret vs. something that's highly specialized like this. Not to mention, keeping up with all the new data, and properly filling in all the metadata could be a real can of worms.
Re: CKAN - The CPAN of data
#13Earlier quoted context omitted.
Hi, I'm one of the CKAN devs. Just wanted to say the site is fully functional again (we've up the cached to be a bit more aggressive). As a side note we have indeed tested with ab :) Our problem is we continue to find the AWS instances we use somewhat unpredictable in their response to load (largely due, we believe, to the fluctuations in CPU "stealing" as load varies across the other instances that share the same ph…
I have a bit of experience with Xen. If you're actually seeing a whole lot of steal (how much?), that's a bad sign because it means you're on a box with a lot of contention. In an ideal world, Xen should steal very little from you. I'm burning all four cores available to me on one of my personal Linodes, and the platform is barely stealing anything. Here's vmstat -s and uptime from that Linode for comparison: 409198…
No disrespect was taken. The hacker news coverage came as a big surprise. We like to turn any caching mostly off and we know this is a risk. This is because we do not want the possibility of any stale data as this annoying to the type of users we have. We are working on a better cache invalidation scheme but this has not been a big priority.
Your feedback is appreciated, thank you.
Edit: Our amount of steal was much much higher than that.
Re: CKAN - The CPAN of data
#14Is this just a rival to Infochimps or is there something different with CKAN that I've missed?
Re: CKAN - The CPAN of data
#15Earlier quoted context omitted.
Hi, I'm one of the CKAN devs. Just wanted to say the site is fully functional again (we've up the cached to be a bit more aggressive). As a side note we have indeed tested with ab :) Our problem is we continue to find the AWS instances we use somewhat unpredictable in their response to load (largely due, we believe, to the fluctuations in CPU "stealing" as load varies across the other instances that share the same ph…
Have you considered implementing some sort of script to scan some of the large biological databases and add links/metadata for the datasets they contain? Looking at what's in CKAN now, it seems that it's mostly datasets that are a bit more easily understood than most of the biological data that's out there, but at the same time indexing and accessing biological data is a HUGE problem for researchers in this field. Th…
Sorry for the late reply. I sadly do not understand the concerns of this field very well. There are many very large datasets referenced on ckan, mainly links to huge triple stores. There are many biological data sets also eg flybase as mentioned. These triple stores are too big to do any decent dynamic linking against which is big shame.
If you get the opportunity could you repost this to ckan-discuss@lists.okfn.org. There are people on that list that understand these issues far more than me and they would love to hear from anyone interested.