OpenStreetMap is doing it with Maps.
I welcome the idea of data being totally free where you make apps to use mirrors instead of APIs
The Web is missing an essential part of infrastructure: an open web index
101–110 of 132 posts
Re: The Web is missing an essential part of infrastructure: an open web index
#102It seems like the idea is recommending the Open Web Index (has its own website). I like a modified version of this. I think that it should be a p2p technology and not try to create one meta-index but rather be many domain-specific ones, with one or more tools or DBs to select which indices to search given a query/context. Are there any decentralized alternatives to Google out there already? I think that also this ove…
we could call searx[1] a decentralized alternative to google or ddg etcetera.
in fact it is an aggregate or meta-search that sends proxy requests to user-selected search engines (with defaults varying from instance to instance).
a list of instances[2] is available via the source git repository. i would recommend a few[3] myself.
as of yet searx does not do some things we might want done:
a) original indexing b) federation between cooperative instances c) offer a spec for archiving data[4]
i think we'll get to something like this soon. there are a lot of pieces in play and it falls to all of us - users, hackers, developers - to participate in development, curate adjoining projects, donate time to test & halcyon & on & on.
as always, it will be interesting to see what we all come up with.
[01.0] https://asciimoo.github.io/searx/ [01.1] searx is copylefted floss via GNU Affero GPL3
[02.0] https://stats.searx.xyz/
[03.0] https://search.disroot.org/ [03.1] this organization respect's EFF's Do Not Track [03.5] https://searx.prvcy.eu [03.6] a secondary useful for reasons indicated by the URI
[04.0] this is where the submitted comes into play. e.g. [04.0] should we develop some sort of open API for domains [04.0] to request archiving? this could take multiple forms [04.0] as a project but as long as it's floss and has RFC...
Re: The Web is missing an essential part of infrastructure: an open web index
#103Earlier quoted context omitted.
This is literally why I created my company: http://www.datastreamer.io/ We've been around for about a decade. IBM watson used us as their social data provider during Jeopardy. We provide data to tons of companies and you're probably using our services - just that it's not obvious where we're used since it's SaaS B2B and not B2C. We're not free but the primary reason we exist is that other vendors charge borderline ex…
If your prices are so much more reasonable than competition, why are they not published publicly on your site? “Contact us and we’ll tell you the price” is shady for a service that claims to be “very very very affordable.”
Because they charge different rates to different people. Super common in b2b arrangements.
Re: The Web is missing an essential part of infrastructure: an open web index
#104Earlier quoted context omitted.
Honestly I'd much rather have a bunch of dice rolls on incompetence than the current centralized, single point of control over the entire index. Google has been purging large swaths of data from the indexes and they won't say how or why or exactly what criteria they are using. It's difficult to imagine a worse solution for the web than this current model.
"Google has been purging large swaths of data from the indexes and they won't say how or why or exactly what criteria they are using." Wow, interesting, this is the first I've heard of this. Might you have some link or citations about this? Thanks.
4% of the Google index hit by de-indexing
https://searchengineland.com/4-of-the-google-index-hit-by-de...
Re: The Web is missing an essential part of infrastructure: an open web index
#105I've always thought it would make more sense if each web server could be responsible for indexing the material that it serves (and offer notifications of updates), so instead of having to crawl everything yourself, you could just request the index from each domain, and then merge them.
This reminds me of how DNS works. Every domain holder is responsible for their nameserver records but every dns server ultimately communicates with a distributed network for lookups. Blockchain would be a great solution for this.
Re: The Web is missing an essential part of infrastructure: an open web index
#106Earlier quoted context omitted.
This doesn't make any sense. You talk about open data but yours is the opposite. You're just another commercial data hoarder, please don't act like you're not.
You are mistaking between free and open. You can be open without being free. Maintaining web index is extremely expensive. Imagine storing most of the web on your own servers and serving it. Someone has to pay bills for all those disk space and bandwidth. I don’t think web index would ever be free (unless storage, compute and bandwidth were free) but having at reasonably priced is a very good thing. I would hope thes…
You are mistaking what an "index" is. You don't need to host any of "the internet" on your servers. You just need to host where it is.
Re: The Web is missing an essential part of infrastructure: an open web index
#107I asked a Google Engineer in a Google Interview (at the end of it, when you get the chance to ask them questions) - if Google would ever make it's infrastructure available to the public so they could leverage it in whatever way they wanted.
He had no idea what I was talking about.
Re: The Web is missing an essential part of infrastructure: an open web index
#108It seems like the idea is recommending the Open Web Index (has its own website). I like a modified version of this. I think that it should be a p2p technology and not try to create one meta-index but rather be many domain-specific ones, with one or more tools or DBs to select which indices to search given a query/context. Are there any decentralized alternatives to Google out there already? I think that also this ove…
Maybe https://www.yacy.net ?
Thanks! I installed it. It seems like exactly the right concept, but the results for the terms that I tested with were horrible.
EDIT: I waited a few minutes and now the results are MUCH better! I think I just needed to let it connect to more peers or something.
Re: The Web is missing an essential part of infrastructure: an open web index
#109How about indexing just the </h1? Is that the intention? We don't want too much information.
I prefer
Re: The Web is missing an essential part of infrastructure: an open web index
#110Could the Internet Archive, specifically https://web.archive.org/ be the basis of an Open Web Index as proposed by the author? I'm sure there are tons of obstacles to that path, but it also would be far ahead of any new initiative in at least two ways: it already has a huge index and ingestion pipeline, and it is a trusted organization.
yes - I worked on this a bit with Mark Graham, the director of the Wayback Machine