Live data from Hacker News

A web crawler that can handle any number of websites

Post reply on HN

Print view View original

1–4 of 4 posts

A web crawler that can handle any number of websites

#1

Post by mixnode » Wed, Apr 12, 2017, 7:48 PM UTC

A web crawler that can handle any number of websites
mixnode.com

Re: A web crawler that can handle any number of websites

#2

Post by spoondocz » Wed, Apr 12, 2017, 8:06 PM UTC

What's the largest crawl you can handle? meaning your upper limit...

Re: A web crawler that can handle any number of websites

#3

Post by mixnode » Wed, Apr 12, 2017, 8:13 PM UTC

spoondocz wrote: post #2

What's the largest crawl you can handle? meaning your upper limit...

We've done crawls of over 250TB and 3 billion web pages. If you need to crawl more than this you can simply shoot us an email and we'll provision resources for your mega crawl.

Re: A web crawler that can handle any number of websites

#4

Post by [deleted] » Wed, Apr 12, 2017, 8:43 PM UTC

[deleted]

Post reply on HN