Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I just finished crawling 5.19B web pages, Ask Me Anything

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 20 posts

Re: I just finished crawling 5.19B web pages, Ask Me Anything

#7

Post by dor_jack »

savethefuture wrote: post #4
savethefuture wrote: post #3

What did you discover.

How long did it take? What type of data did you record?

It took us about 13 days. We recorded reources of all types: text/, image/, application/*

As one would expect the vast majority of data recorded is text/* (html,...)

Re: I just finished crawling 5.19B web pages, Ask Me Anything

#8

Post by savethefuture »

dor_jack wrote: post #6
savethefuture wrote: post #3

What did you discover.

We are processing the data as we speak. However the movement of technology based on where your company is based is truly incredible. Will update this in a few days with more data.

That will be an interesting correlation to see different frameworks or tech or even design elements based on geographical location.

Re: I just finished crawling 5.19B web pages, Ask Me Anything

#9

Post by dor_jack »

savethefuture wrote: post #8
dor_jack wrote: post #6

Earlier quoted context omitted.

We are processing the data as we speak. However the movement of technology based on where your company is based is truly incredible. Will update this in a few days with more data.

That will be an interesting correlation to see different frameworks or tech or even design elements based on geographical location.

If our company approves I would like to publish some general statistics that may be of interest to others.
Post reply on HN