Viewing profile — outpan

Outpan here, We use a combination of polite massive-scale web crawling and user contribution. We had an in-house web crawler for years until recently when we released it as a stand…

Semantic web is too good of an idea not too iterate on constantly :)

haha. We should use this as the project mantra.

Would you be open to having a little chat via email? hi@outpan.com

This is great! thanks. I will look into adding the dbpedia.org data.

"a journey of a thousand miles begins with a single step"

As for your question about real world applications: Outpan was only a product database up until last week. It is used in over a hundred apps, some with more than a million users. I…

I think graph overlap is what actually determines what data is "accurate". There are currently bots writing to the database by people who are not connected and I have yet to see th…

I was driving so couldn't stay on the phone. The number of keys for different concepts is just very large (if not infinite for the sake of avoiding philosophical debates). It will …

Up until last week, outpan was strictly used for gathering data on product barcodes. We are in the process of adding data in other categories.

Twitter url seems like a strong key since it is referenced in many other contexts. As for curation, it is intended to provide examples of what key, attr and values are regardless o…

Common Crawl is great! however, some use cases require larger crawls with a higher frequency.

Thanks a lot! This sounds reasonable. Did you guys look into professional services for this?

Would you be able to share what your stack was? and the resources it took? Thanks a lot.

I'm not sure how he manages to crawl with this speed using such low amount of resources. We did a benchmark on Nutch and couldn't really pass the 10-14 M(B)ps on a $1200/month mach…

That post is what triggered my Ask post. The problem is the huge contrast with https://www.quora.com/How-much-would-it-cost-to-crawl-1-bill... Even taking into account the drop in …

I'm really curious to find out how much it'll cost to crawl a billion pages. Doesn't really matter if you used a SaaS solution or built your own crawler, any info would be really u…

Awesome job! For the life of me I can't figure out how you manage to crawl over a billion web pages (even in 2-3 months), index the data and run the server with €300 per month. Esp…

Up to 2000 calls/minute. We will hopefully increase this soon.

That's true; the mandatory sing-up is a temporary measure, I'm sure there are a lot of ways to fight spammers including the method you mentioned. Thank you for your suggestion.