21–30 of 132 posts
Earlier quoted context omitted.
Doesn't matter. A global index that anyone can then use to further process would be very helpful in making a non-profit alternative to google.
Only if the index collects all the necessary (meta)data for your application. You can't get additional data by post-processing.
I assume that in a world of competitive index users, there is no one size fits all. Presumably application design (and feature) choices will heavily influence how the index should work. For simple "I know TF-IDF, let's build a toy search engine" it will suffice, but apart from that?
This is the problem with this idea. Format of the index will be intricately tied to the algorithms that are meant to traverse it. The production of a search result by Google or Bing in a fraction of a second is an outright miracle of software engineering. If this open index service provides something developers can easily understand and consume, such as a term-doc hitlist with a simple encoding, it will be enormous,…
Some users may be happy to wait for hours or days to get high quality answers not available from commercial companies. Can still be faster than emailing a human friend or consultant or tasking an employee or department.
While I like the idea, I fear the potential for abuse, conflict and community splits. It will need some sort of moderation, at least to prevent: 1. spam 2. child pornography 3. content against the laws The only thing that is easy to define as policy is #2. No one likes child porn. But even then, there are grey areas with differing legal status - lolicon on the anime side and "barely legal" on the realistic side, plus…
I hope you do realize the contradiction.
I have been wanting this for years... If you look at the original Yahoo Page when Yahoo first started out it attempted to solve this problem. I believe this index could be regionally or language based... In the United States one could use Dewey Decimal https://en.wikipedia.org/wiki/Dewey_Decimal_Classification Library of Congress https://en.wikipedia.org/wiki/Library_of_Congress_Classifica...
It won't work without a central authority. See Soundcloud as an example. People tag their music with whatever they think will get them traffic. So, in order to do this you'll need a mass of volunteers which will lead to politics "XYZ should be classified as G! No it should be F!", "classifying ABC as DEF is racists/sexist/..." and other arguments. You'll also get people lobbying to have things removed (right to be fo…
Our project LearnAwesome[1] currently relies on volunteers to curate topics, but classfication / ontology engineering is in fact seems to be a hard problem.
Common Crawl (non-profit): Stores regular, broad, monthly crawls as WARC files. Provides a separate index that can be used to look data up (no a fulltext index though). Used mostly in academia.
Mixnode (for-profit): Regularly crawls the web and lets users write SQL queries against the data. Not sure who the primary users are since it's in private beta.
There are some search engine APIs, but I don't think the conflict of interest would allow for cost-effective large-scale access and pricing...
Regardless, the notion of a general web index is well nigh moot at this point due to its not having been built into the system from the get-go. Any such attempt at this point will be, by definition, ad hoc and built by some group of individuals, with the vastness of the content, the cost of the project and the intrinsic conflicts that will no doubt arise making independence from finance and legal issues non-trivial, to say the least.
Really, Wikipedia is the most sensible foundation I can imagine, given that Google has become a self-serving for-profit corporate advertising machine.
Earlier quoted context omitted.
This is the problem with this idea. Format of the index will be intricately tied to the algorithms that are meant to traverse it. The production of a search result by Google or Bing in a fraction of a second is an outright miracle of software engineering. If this open index service provides something developers can easily understand and consume, such as a term-doc hitlist with a simple encoding, it will be enormous,…
Google needs sub-second response to show ads. Some users may be happy to wait for hours or days to get high quality answers not available from commercial companies. Can still be faster than emailing a human friend or consultant or tasking an employee or department.
Earlier quoted context omitted.
Google needs sub-second response to show ads. Some users may be happy to wait for hours or days to get high quality answers not available from commercial companies. Can still be faster than emailing a human friend or consultant or tasking an employee or department.
No idea where you got this statisti, but I guarantee no user would be happy to wait hours or (god forbid) days on a search result.
Central search indexes like Google are not going away. There are client-side metasearch interfaces that combine Google results with other sources. Those other sources can be much slower, including human responses. You would still have your synchronous sub-second response from centralized search, but there would be asynchronous results from decentralized search.
This exists today, e.g. when you post a question on HN or a messaging app, asking other humans for answers not available in public indexes. Most of the world's knowledge is not public, it's obscure and may only be of interest to specific niche audiences.