Ask HN: Can we create a new internet where search engines are irrelevant? • Hacker News Board

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#341

Post by bduerst » Wed, Jun 26, 2019, 8:26 PM UTC

_nalply wrote: post #14

The deeper problem is advertising. It is sort of a prisoner's dilemma: all commercial entities have a shouting contest to attract customer attention. It's expensive for everybody. If we could kill advertisement permanently, we can have an internet as described in the question. This will almost be like an emergent feature of the internet.

The only way to kill advertising is to have perfectly efficient markets.

Until then, you're going to have demand for ferrying information between sellers and buyers, and vice versa, because of information asymmetry. You may disagree with some of the mediums currently used, finding them annoying, but advertising is always evolving to solve this problem, as is evident in the last three decades.

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#342

Post by pbhjpbhj » Wed, Jun 26, 2019, 8:29 PM UTC

pavas wrote: post #203

pbhjpbhj wrote: post #182

Earlier quoted context omitted.

>We have a pretty good idea of which people are trustworthy (or capable, or dependable, or any other characteristic) in our daily lives // We really don't. People get surprised all the time that someone had an affair, or cheated, or ripped someone off, or whatever. "But I trusted you" ... It's actually relatively easy to fool people in to trusting you, as many red team members will probably confirm. Look at someone l…

So how do _you_ make any sort of judgments based off of what people say? What information do you use to judge whether their statements are accurate? Or do you always start with the assumption that everything everyone says is suspect? What sort of information do you use to come to any sort of conclusion, and how do you determine the trustworthiness of that information? >This is domain authority again - trust some doma…

Pyrrhonism, you start on the assumption that no-one [else] even exists and go from there ... ;o)

Seriously, I'm not so sure -- I try to trust first and then update that status as more information becomes available; but that's more of a religious position.

I don't think it's necessarily instructive to look at my personal modes here. I guess my main point is that if you're going to say "well humans have cracked trust, we'll just model it on that" then I think you're shooting wide of the mark.

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#343

Post by ken » Wed, Jun 26, 2019, 8:33 PM UTC

dymk wrote: post #204

ken wrote: post #172

Earlier quoted context omitted.

The smarts living on-device is not necessarily the same as the smarts executing on-device. We already have the means to execute arbitrary code (JS) or specific database queries (SQL) on remote hosts. It's not inconceivable, to me, that my device "knowing me" could consist of building up a local database of the types of things that I want to see, and when I ask it to do a new search, it can assemble a small program wh…

I think you're suggesting homomorphic encryption to execute the user's ranking model. Unfortunately, homomorphic encryption is pretty slow, and the types of operations you can do are limited. But it's viable if the data you're operating on is relatively small - e.g. just searching through (encrypted) personal messages or something.

I think you've got the right general idea, but I don't know that it has to be homomorphic encryption. After all, an index of the public web is not really secret, and the user doesn't have a private key for it.

In the simplest case, you could make a search engine in the form of a big, public, regularly-updated database, and let users send in arbitrary queries (run in a sandbox/quota environment).

That's essentially what we've got now, except the query parser is a proprietary black box that changes all the time. I don't see any inherent reason they couldn't expose a lower-level interface, and let browsers build queries. Why can't web browsers be responsible for converting a user's text (or voice) into a search engine query structure?

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#344

Post by samirm » Wed, Jun 26, 2019, 8:43 PM UTC

aleppe7766 wrote: post #333

mavsman wrote: post #287

Earlier quoted context omitted.

How about open sourcing the ranking and then allowing people to customize it. I should be able to rank my own search results how I want to without much technical knowledge. I want to rank my results by what is most popular to my friends (Facebook or otherwise) so I just look for a search engine extension that allows me to do that. This could get complex but can also be simple if novices just use the most popular rank…

That would bring to an even bigger filter bubble issue, more precisely to a techno élite which is capable, willing and knowledgeable enough to feel the need go through the hassle, and all the rest navigating in such an indexed mess that would pave the way to all sort of new gatekeepers, belonging to the aforementioned tech élite. It’s not a simple issue to tackle, perhaps a public scrutiny on the ranking algorithms w…

I disagree. The people who don't know anything and are unwilling to learn wouldn't be any worse off than they are today and everyone else would benefit from an open source "marketplace" of possible ranking algorithms that the so called "techno elite" have developed.

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#345

Post by abathur » Wed, Jun 26, 2019, 8:51 PM UTC

pharke wrote: post #258

TheOtherHobbes wrote: post #219

Earlier quoted context omitted.

This would be ludicrously easy to game. Crowdsourcing would also be ludicrously easy to game. The problem isn't solvable without a good AI content scraper. The scraper/indexer either has to be centralised - an international resource run independently of countries, corporations, and paid interest groups - or it has be an impossible-to-game distributed resource. The former is hugely challenging politically, because the…

Just want to point out that you're on a site that successfully uses crowd sourcing combined with moderation to curate a list of websites, news, and articles that people find interesting and valuable. Why not a new internet built around communities like this where the users actively participate in finding, ranking, and moderating the content they consume? It's not a stretch to add a decent search index and categories…

Edit: I had myself convinced that comments have a different ID space from submissions, but that obviously isn't true. I've partly rewritten to correct for an over-guess on how many new submissions there are each day.

I agree with your general suggestion, but just want to highlight that scale issues still make me think whatever finds traction on HN is a bit of a crapshoot.

It looks like there were over 10k posts (including comments) in the last day, and the list of submissions that spent time on the front page day yesterday has 84 posts. I don't how normal the last 2 days were, but by eyeball I'd guess around a quarter of the posts are comments on the day's front-page posts. This means there are probably a few thousand submissions that didn't get much if any traction.

Any time I look at the "New" page, I still end up finding several items that sound interesting enough to open. I see more than 10 that I'm tempted to click on right now. The current new page stretches back about 40 minutes, and only 10 of the 30 have more than 1 point (and only 1 has more than 10). Only 2 of the links I was tempted to click on have more than 1 point.

I suspect that there's vastly more interesting stuff posted to HN than its current dynamics are capable of identifying and signal-boosting. That's not bad, per se. It'd be an even worse time-sink if it were better at this task. But it does mean there are pitfalls using it as a model at an even larger scale and in other contexts.

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#346

Post by Theodores » Wed, Jun 26, 2019, 8:57 PM UTC

penagwin wrote: post #311

Retra wrote: post #278

Earlier quoted context omitted.

Couldn't you just randomize result ordering?

You know how google search results can get really useless just a few pages in? And it says it found something crazy like 880,000 results? Imagine randomizing that. --- Unrelated I searched for "Penguin exhibits in Michigan". Of which we have several. It reports 880,000 results but I can only go to page 12 (after telling it to show omitted results). Interesting... https://www.google.com/search?q=penguin+exhibits+in+mi…

If you think of it as like an old fashioned library or an old fashioned Blockbuster video store.

Sure you could read any book ever printed in the English language in the local library. They might have to get it in from the national collection or the big library in the city. But you ain't going to see every book in the local library. There is more than you could wish for and you will never read every book in the local library. But all the classics are there, the talked about new books are there (or out on loan, back soon). All the reference books that school kids are there, there is enough to get you started in any hobby.

Google search results are like that. Those 880,000 'titles' are a bit like the Library of Congress boasting how big it is, it is just a number. All they have really got for you is a small selection that is good enough for 99% of people 99% of the time. Only new stuff by people with Page rank (books with publishers) get indexed now and put into the 'main collection'.

Much like how public libraries do have book sales, Google do let a lot of the 880,000 results drop off.

It's a ruse!

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#347

Post by z3t4 » Wed, Jun 26, 2019, 8:59 PM UTC

rhmw2b wrote: post #317

quelsolaar wrote: post #261

Yes, we need search engines, but they don't need to be monolithic. Imagine that indexing the text of your average web page takes up 10k. Then you get 100.000 pages per Gig. It means that you if you spend ~270USD on a consumer 10 tera drive you can index a billion webpages. Google no longer says how many pages they index, but its estimated to be with in one order of magnitude of that. This means that in terms of hardw…

Google's paper on Percolator from 2010 says there are more than 1T web pages. 9 years later there is surely way more than that. https://ai.google/research/pubs/pub36726 The real issue would be crawling and indexing all those pages. How long would it take for an average user's computer with a 10Mb internet connection to crawl the entire web? It's not as easy a problem as you make it seem.

Could be done with a p2p "swarm". Peers get asigned pages to index then share the result.

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#348

Post by timlatim » Wed, Jun 26, 2019, 9:09 PM UTC

Kaiyou wrote: post #56

bpye wrote: post #29

Earlier quoted context omitted.

And then 5 other users ask the same question because they have no search engine. I think this gets boring quick...

Teachers teaching the same class every year can't use this as an excuse either.

The most informative answers I've encountered on StackOverflow are either a product of research (benchmarking, analyzing multiple sources) or very specific knowledge, sometimes written by the author of the framework/library in question. I'm not sure your analogy applies since these answers demand substantially more effort than the (usually) predictable and repetitive questions teachers face in class.

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#349

Post by basch » Wed, Jun 26, 2019, 9:13 PM UTC

abathur wrote: post #339

basch wrote: post #238

Earlier quoted context omitted.

i disagree that there isnt a way, just that nobodies tried a good one yet. take reddit for example. it should be very easy to establish a few voters who make "good" decisions, and then extrapolate their good decisions based on people with similar voting patterns. it would combine a million monkeys with typewriters with expert meritocracy. you want different sorting, sort by different experts until you get the results…

It should, but if anyone knows who these kingmakers are, it's still probably just a matter of time before they accrue enough power for it to be worth someone's time to at least try to track them down and manipulate their decisions (bribe, blackmail, sponsor, send free trials, target with marketing/propaganda campaigns, etc.)

Who says it even has the same kingmakers every day? Slashdot solved that part of metamoderation two decades ago.

A person might be an expert in cars but not horses. A car expert might be superseded . The seed data creators could be a fluid thing.

Re: Ask HN: Can we create a new internet where search engines are irrelevant?

#350

Post by kickscondor » Wed, Jun 26, 2019, 9:13 PM UTC

brokensegue wrote: post #300

kickscondor wrote: post #233

Earlier quoted context omitted.

Directories are still useful - Archive of Our Own ( https://archiveofourown.org/ ) is a large example for fan fiction, Wikipedia has a full directory ( https://en.wikipedia.org/wiki/Category:Main_topic_classifica... ), Reddit wikis perform this function, Awesome directories ( https://github.com/sindresorhus/awesome ) or personal directories like mine at href.cool. The Web is too big for a single large directory - but…

Ao3 isn't really a directory since they do the actual hosting

Yes, thank you - I only mean in terms of organization.