41–50 of 132 posts
I didn't see mention of who would pay for this infrastructure. Is it considered a gov't funded or volunteer / donation thing? There doesn't seem to be a mention of how to alleviate a tragedy of the commons problem (unless I missed it). If common crawl is doing a fine job, who funds them?
While I like the idea, I fear the potential for abuse, conflict and community splits. It will need some sort of moderation, at least to prevent: 1. spam 2. child pornography 3. content against the laws The only thing that is easy to define as policy is #2. No one likes child porn. But even then, there are grey areas with differing legal status - lolicon on the anime side and "barely legal" on the realistic side, plus…
The proposal is for a publicly funded index as base-level infrastructure. Filtering out spam, pornography, and other undesirable or illegal content would be done at the service level, i.e., but companies/organizations building user-facing search applications on top of the index.
There are two entities trying to pull this off: Common Crawl (non-profit): Stores regular, broad, monthly crawls as WARC files. Provides a separate index that can be used to look data up (no a fulltext index though). Used mostly in academia. Mixnode (for-profit): Regularly crawls the web and lets users write SQL queries against the data. Not sure who the primary users are since it's in private beta. There are some se…
Not for existing search machine providers, but I think there is room for new players to do this large scale. Imagine an AWS service that high performance access to crawled data as well as a number of indexes and a fairly simple search engine using this data. That would commoditize one of Google's biggest advantages, and anyone could, at least in principle, run their own search engine from the data. Because the market for this is much wider than traditional search engines just providing the data and indices for a pay-as-you-go fee could still be very profitable.
I've always thought it would make more sense if each web server could be responsible for indexing the material that it serves (and offer notifications of updates), so instead of having to crawl everything yourself, you could just request the index from each domain, and then merge them.
I've always thought it would make more sense if each web server could be responsible for indexing the material that it serves (and offer notifications of updates), so instead of having to crawl everything yourself, you could just request the index from each domain, and then merge them.
While I like the idea, I fear the potential for abuse, conflict and community splits. It will need some sort of moderation, at least to prevent: 1. spam 2. child pornography 3. content against the laws The only thing that is easy to define as policy is #2. No one likes child porn. But even then, there are grey areas with differing legal status - lolicon on the anime side and "barely legal" on the realistic side, plus…
The proposal is for a publicly funded index as base-level infrastructure. Filtering out spam, pornography, and other undesirable or illegal content would be done at the service level, i.e., but companies/organizations building user-facing search applications on top of the index.
No, it has to be done before, at the infrastructure level. There are jurisdictions (Germany, for one!) where even the storage or publication of links can be illegal under certain circumstances. With the new GDPR law and whatever is coming up in the US, the situation is even more unclear as it is trivial to embed protected personal data into URLs.
I've always thought it would make more sense if each web server could be responsible for indexing the material that it serves (and offer notifications of updates), so instead of having to crawl everything yourself, you could just request the index from each domain, and then merge them.
Like a sitemap? https://en.m.wikipedia.org/wiki/Sitemaps
Earlier quoted context omitted.
I dont believe Common Crawl offers a real time search index as its delayed by more than a month (although that could have changed recently). Still useful for research purposes but not that desirable for a search engine that competes with Google, etc.
Would users notice for many searches? Obviously it wouldn't be useful for news or social media, but for practically everything else a latency of a month would be fine.
While you say "news" really that covers any information about current-ish events. It's not just "what happened today" but background on things like the Muller report right now.
Any technology release, or update.
Reviews of any hardware or software.
Information about security vulnerabilities.
Film reviews.
Game reviews.
Book reviews.
New scientific publications.
Isn't this what Common Crawl[1] is. From their FAQ: > What is Common Crawl? > Common Crawl is a 501(c)(3) non-profit organization dedicated to providing a copy of the internet to internet researchers, companies and individuals at no cost for the purpose of research and analysis. > What can you do with a copy of the web? > The possibilities are endless, but people have used the data to improve language translation sof…