331–340 of 395 posts
For example, if you build on a decentralized network, ask yourself how you can prevent SEO companies from adding a huge amount of nodes to promote certain sites.
The 2 core flaws of the Internet (more precisely the World Wide Web) are lack of native search and native payments. Cryptocurrencies have started to address the second issue, but no one that I know of is seriously working on the first. Fast information retrieval requires an index. A better formulation of the question might be: how do we maintain a shared, distributed index that won't be destroyed by bad actors. I won…
It's an impossible problem to solve because we don't have good consistent metadata to draw on. Libraries work because they have good metadata to catalog their collections. Good metadata needs to be generated by hand, doing it automatically is bound to lead to errors and special cases that will pollute your search results.
I say we abandon the idea of the ideal search engine, accept the fact that we will never be able to find every needle in every haystack, and defer to a decentralized assortment of thousands of topic-specific indexes of relevant information. Some of them will be shit, but that's fine, the internet has always been a refuge for conspiracy theorists and other zaney interests. The good stuff will shine through the mud, as it's always done.
I think it would be helpful to remember to distinguish two separate search engine concepts here: indexing and ranking. Indexing isn't the source of problems. You can index in an objective manner. A new architecture for the web doesn't need to eliminate indexing. Ranking is where it gets controversial. When you rank, you pick winners and losers. Hopefully based on some useful metric, but the devil is in the details on…
How about open sourcing the ranking and then allowing people to customize it. I should be able to rank my own search results how I want to without much technical knowledge. I want to rank my results by what is most popular to my friends (Facebook or otherwise) so I just look for a search engine extension that allows me to do that. This could get complex but can also be simple if novices just use the most popular rank…
'Web of trust' has its flaws too: a sufficiently large number of malicious nodes cooperating can subvert the network.
However, maybe we can exploit locality in the graph? If the user has an easy way to indicate the quality of results, and we cluster the graph of relevance sources, the barrier to subverting the network can be raised significantly.
Let's say that each ranking server indicates 'neighbours' which it considers relatively trustworthy. When a user first performs a search their client will pick a small number of servers at random, and generate results based on them.
* If the results are good, those servers get a bit more weight in future. We can assume that the results are good if the user finds what they're looking for in the top 5 or so hits (varying depending on how specific their query is; this would need some extra smarts).
* If the results are poor (the user indicates such, or tries many pages with no luck) those servers get downweighted.
* If the results are actively malicious (indicated by the user) then this gets recorded too...
There would need to be some way of distributing the weightings based on what the servers supplied, too. If someone's shovelling high weightings at us for utter crap, they need to get the brunt of the downweighting/malice markers.
Servers would gain or lose weighting and malice based on their advertised neighbours too. Something like PageRank? The idea is to hammer the trusting server more than the trusted, to encourage some degree of self-policing.
Users could also chose to trust others' clients, and import their weighting graph (but with a multiplier).
Every search still includes random servers, to try to avoid getting stuck in an echo chamber. The overall server graph could be examined for clustering and a special effort made to avoid selecting more than X servers in a given cluster. This might help deal with malicious groups of servers, which would eventually get isolated. It would be necessary to compromise a lot of established servers in order to get enough connections.
Of course, then we have the question of who is going to run all these servers, how the search algorithm is going to shard efficiently and securely, etc etc.
Anyone up for a weekend project? >_>
To get rid of search engines like Google and Bing we don't need to build a new internet - we just need to build new search engines. E.g., how about an open source spider/crawler that anyone can run on their own machine continuously contributing towards a distributed index that can be queried in a p2p fashion. (Kind of like SETI@home but for stealing back the internet). Just think about all the great things that resea…
You don't remember how it was before search engines, do you? It was like a dark maze, and sometimes you'd find a piece of the map. Search coming online was a watershed moment -- like, "before search" and "after search"
- You had your web rings, which would cycle from site to site based on a category, some pages having multiple rings.
- You had your "communities", organizing sites by URL structure, where similar pages were grouped together like a strip mall or something (i.e. neighborhoods for geocities).
- You had scammy services that would submit your pages to multiple search engines, at a cost, but would guarantee you would show up in results.
- You had your aggregators, like dogpile, where you would sift through pages of results from different search engines, hoping to find something different.
It wasn't a good time. If you think about the problem that search engines solve today - connecting people with information that they want - we're currently at a peak.
I think it would be helpful to remember to distinguish two separate search engine concepts here: indexing and ranking. Indexing isn't the source of problems. You can index in an objective manner. A new architecture for the web doesn't need to eliminate indexing. Ranking is where it gets controversial. When you rank, you pick winners and losers. Hopefully based on some useful metric, but the devil is in the details on…
Earlier quoted context omitted.
The smarts living on-device is not necessarily the same as the smarts executing on-device. We already have the means to execute arbitrary code (JS) or specific database queries (SQL) on remote hosts. It's not inconceivable, to me, that my device "knowing me" could consist of building up a local database of the types of things that I want to see, and when I ask it to do a new search, it can assemble a small program wh…
To some extent, doesn't Google already do this? Meaning that based on your location/Google account/other factors such as cookies or search history, it will tailor your results. For instance, searching the same query on different computers will result in different results. Though to your point, google probably ends up storing this information in the cloud
Earlier quoted context omitted.
This. Everyone's missing the point of a search engine. We're talking about billions of pages and if not ranked (authority is a good hueristic), filtered (de-ranked), etc then good luck finding valuable information because everyone is gaming the systems to improve their ranking. I think this is part of the reason you get a lot of fake news on social media. It's a constant stream of information (a new dimension of time…
i disagree that there isnt a way, just that nobodies tried a good one yet. take reddit for example. it should be very easy to establish a few voters who make "good" decisions, and then extrapolate their good decisions based on people with similar voting patterns. it would combine a million monkeys with typewriters with expert meritocracy. you want different sorting, sort by different experts until you get the results…
That was what the early internet was like (I was there). People built indexes by hand, lists of pages on certain topics. There was the Gopher protocol that was supposed to help with finding things. But this was all top-down stuff, the first indexing/crawling search engines were bottom-up and it worked so much better. And for a while we had an ecosystem of different search engines until Google came along, was genuinel…
Search is the problem. If you don’t rank in google you don’t exist on the internet. There is an entire economy built on manipulating search that is pay to play in addition to google continually focusing on paid search of natural SERPs. Controlling search right now is controlling the internet.
Maybe in 2009. Today there are businesses today that exist solely on Instagram, Facebook, Amazon, etc.