11–15 of 15 posts
Earlier quoted context omitted.
Would you be able to share what your stack was? and the resources it took? Thanks a lot.
Ruby and Sidekiq as the messaging queue Postgres to store the data Elasticsearch as a search index. My ES cluster has around 10 nodes, 64 GB RAM, quad-core. Postgres database cluster is 4 nodes, 1 TB, 64 GB RAM, quad-core. 800 crawler threads distributed across 10 dedicated servers.
Earlier quoted context omitted.
Would you be able to share what your stack was? and the resources it took? Thanks a lot.
Just in case you don't know common-crawl makes available a huge crawl dataset
Earlier quoted context omitted.
Ruby and Sidekiq as the messaging queue Postgres to store the data Elasticsearch as a search index. My ES cluster has around 10 nodes, 64 GB RAM, quad-core. Postgres database cluster is 4 nodes, 1 TB, 64 GB RAM, quad-core. 800 crawler threads distributed across 10 dedicated servers.
Thanks a lot! This sounds reasonable. Did you guys look into professional services for this?
The real innovation won't be in crawling but in working on the index, filtering it, organizing it, trying sort algorithms and learning.
If this was available and gained popularity I could see competition in search again.
I've crawled over a billion pages over a stretch of 3 years or so. Crawling is the easy task and just crawling a billion pages wouldn't cost more than a few thousand a month. Add a couple more thousand for storing these pages in a search index and database.
Thanks!