Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CKAN - The CPAN of data

ckan.net

Post reply on HN
Print view View original

1–10 of 15 posts

Re: CKAN - The CPAN of data

#3

Post by jedsmith »

Every time something interesting that I want to read shows up on the front page, and clicking on it yields a spinning tab while the other end flaps about in agony, I can't help but wonder how many administrators know how to use ab(8). The rate of dead sites on HN is really a shocker, since commodity cloud that can stand up to HN's load -- as opposed to the shared hosting of yesterdecade -- is so widely available. I would have thought that Slashdotting would be a historical problem by now...

Is it not common sense to test the hell out of something before someone who would submit it to HN is even aware of its existence? WordPress is pretty bad for this (since most admins follow the directions and don't bother tweaking), and I've heard that Drupal can be too. Without tweaking and caching stuff, you'll fall over quick in front of this many eyeballs.

I know, easy for me to say.

Re: CKAN - The CPAN of data

#4

Post by nprincigalli »

jedsmith wrote: post #3

Every time something interesting that I want to read shows up on the front page, and clicking on it yields a spinning tab while the other end flaps about in agony, I can't help but wonder how many administrators know how to use ab(8). The rate of dead sites on HN is really a shocker, since commodity cloud that can stand up to HN's load -- as opposed to the shared hosting of yesterdecade -- is so widely available. I w…

On google's cache: http://webcache.googleusercontent.com/search?q=cache:yNyANy-...

And I bet the poster (ig1) is not the site's admin.

Re: CKAN - The CPAN of data

#5

Post by wwaites »

jedsmith wrote: post #3

Every time something interesting that I want to read shows up on the front page, and clicking on it yields a spinning tab while the other end flaps about in agony, I can't help but wonder how many administrators know how to use ab(8). The rate of dead sites on HN is really a shocker, since commodity cloud that can stand up to HN's load -- as opposed to the shared hosting of yesterdecade -- is so widely available. I w…

seems to be back up now... the http headers suggest caching with a time of >30 mins is now in effect... i think caching and cache poisoning is generally hard to get right for dynamic data-driven sites...

Re: CKAN - The CPAN of data

#6

Post by kindly »

jedsmith wrote: post #3

Every time something interesting that I want to read shows up on the front page, and clicking on it yields a spinning tab while the other end flaps about in agony, I can't help but wonder how many administrators know how to use ab(8). The rate of dead sites on HN is really a shocker, since commodity cloud that can stand up to HN's load -- as opposed to the shared hosting of yesterdecade -- is so widely available. I w…

Hi, I'm one of the CKAN devs. Just wanted to say the site is fully functional again (we've up the cached to be a bit more aggressive).

As a side note we have indeed tested with ab :) Our problem is we continue to find the AWS instances we use somewhat unpredictable in their response to load (largely due, we believe, to the fluctuations in CPU "stealing" as load varies across the other instances that share the same physical box).

Anyway, if you want to know more about ckan have a look on http://ckan.org/about.

Re: CKAN - The CPAN of data

#7

Post by kindly »

nprincigalli wrote: post #4
jedsmith wrote: post #3

Every time something interesting that I want to read shows up on the front page, and clicking on it yields a spinning tab while the other end flaps about in agony, I can't help but wonder how many administrators know how to use ab(8). The rate of dead sites on HN is really a shocker, since commodity cloud that can stand up to HN's load -- as opposed to the shared hosting of yesterdecade -- is so widely available. I w…

On google's cache: http://webcache.googleusercontent.com/search?q=cache:yNyANy-... And I bet the poster (ig1) is not the site's admin.

You are correct.

Re: CKAN - The CPAN of data

#9

Post by jedsmith »

kindly wrote: post #6
jedsmith wrote: post #3

Every time something interesting that I want to read shows up on the front page, and clicking on it yields a spinning tab while the other end flaps about in agony, I can't help but wonder how many administrators know how to use ab(8). The rate of dead sites on HN is really a shocker, since commodity cloud that can stand up to HN's load -- as opposed to the shared hosting of yesterdecade -- is so widely available. I w…

Hi, I'm one of the CKAN devs. Just wanted to say the site is fully functional again (we've up the cached to be a bit more aggressive). As a side note we have indeed tested with ab :) Our problem is we continue to find the AWS instances we use somewhat unpredictable in their response to load (largely due, we believe, to the fluctuations in CPU "stealing" as load varies across the other instances that share the same ph…

I have a bit of experience with Xen. If you're actually seeing a whole lot of steal (how much?), that's a bad sign because it means you're on a box with a lot of contention. In an ideal world, Xen should steal very little from you. I'm burning all four cores available to me on one of my personal Linodes, and the platform is barely stealing anything. Here's vmstat -s and uptime from that Linode for comparison:

       409198 non-nice user cpu ticks
     60878563 nice user cpu ticks
       166987 system cpu ticks
    811571786 idle cpu ticks
      4486779 IO-wait cpu ticks
           25 IRQ cpu ticks
        15388 softirq cpu ticks
       766577 stolen cpu ticks

    12:06:10 up 13 days, 14:11,  3 users,  load average: 4.00, 4.01, 4.05
I've had the pedal to the floor for a couple of days on the CPU, and only 766 kticks have been stolen (total) since I booted. If you're seeing a lot more steal than that, your host is working pretty hard to schedule the domUs fairly.

Wouldn't dare to assume that I know better how to run operation than you do, just sharing my experiences with Xen. Netflix had a solution to this -- unfortunate that it was necessary, but a solution nonetheless -- which was to monitor steal closely and spin up a new instance if it skyrocketed: http://blog.sciencelogic.com/netflix-steals-time-in-the-clou...

Given the opportunity, I'd like to point out that I meant no disrespect in my original comment, if it wasn't clear. I was speaking more from a generality and not about CKAN specifically, a fact lost on those mindlessly downvoting me.

Re: CKAN - The CPAN of data

#10

Post by jedsmith »

nprincigalli wrote: post #4
jedsmith wrote: post #3

Every time something interesting that I want to read shows up on the front page, and clicking on it yields a spinning tab while the other end flaps about in agony, I can't help but wonder how many administrators know how to use ab(8). The rate of dead sites on HN is really a shocker, since commodity cloud that can stand up to HN's load -- as opposed to the shared hosting of yesterdecade -- is so widely available. I w…

On google's cache: http://webcache.googleusercontent.com/search?q=cache:yNyANy-... And I bet the poster (ig1) is not the site's admin.

I never spoke to ig1 in my comment. "You'll" there was in the general.
Post reply on HN