Viewing profile — dor_jack

If you need to perform a web-scale crawl I strongly recommend https://www.mixnode.com .

The platform we used provided their own seedlist and took it from there.

It was an all inclusive deal: 420 TB at 0.06 per GB = $25,804

If our company approves I would like to publish some general statistics that may be of interest to others.

It took us about 13 days. We recorded reources of all types: text/ , image/ , application/* As one would expect the vast majority of data recorded is text/* (html,...)

We are processing the data as we speak. However the movement of technology based on where your company is based is truly incredible. Will update this in a few days with more data.

Our company is in the Marketing Intelligence (MI) industry. We needed to measure the penetration of multiple technologies in different countries.

I WAS JUST RATE LIMITED BY HN, SO IM GOING TO ANSWER YOUR QUESTIONS UNDER A NEW ACCOUNT: dor_jack_2