Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Made Google's "Web" View My Default Search

tedium.co

Post reply on HN
Print view View original

111–120 of 154 posts

Re: How I Made Google's "Web" View My Default Search

#111

Post by chongli »

lelanthran wrote: post #96
chongli wrote: post #93

Earlier quoted context omitted.

There’s quite a lot of dark patterns to watch out for. One is including text on a page meant to be read by your crawler but which is hidden from users by various CSS tricks. To get around this you’d either need to specifically blacklist certain CSS / DOM structures or do something very radical: render the page and then attempt to OCR it back into text to match against the original text.

> One is including text on a page meant to be read by your crawler but which is hidden from users by various CSS tricks. Shouldn't matter too much, I think. If the score starts off perfect, and can only go down with each thing found (i.e. get penalised), then the problem with adding human-invisible content is irrelevancy, whcih you are letting humans score anyway.

Humans aren’t going to be there to score your content for you at the beginning. You need to deliver useful results first.

Re: How I Made Google's "Web" View My Default Search

#112

Post by immibis »

Aachen wrote: post #97
nicce wrote: post #44

Earlier quoted context omitted.

> allow logged-in users to down-rank a site, etc. And then you need a huge anti-bot mechanism if the search engine will get any popularity. There is money-based motivation to affect the rankings.

I keep thinking of a hierarchical invite model for these sorts of problems. I'd be very curious if someone could give a second opinion on this idea. The mechanism: Everyone has to be invited by someone, so it all traces back to the creator. The creator knows they themselves are legit, but let's say someone online asked for an invite and bad inputs keep coming from somewhere down that branch of the invite tree. Either…

This is the traditional way societies enforce totalitarianism. You're not allowed to sleep outside, because your landlord will punish you. He has to, because if he doesn't, the city will punish him. It has to, because if it doesn't, the state will punish it.

Maybe totalitarianism is what you want for search engine moderation. But it does feel a bit messed up when you frame it like I just did. Another thing that happens in these power delegation hierarchies is that people undermine each other in order to move up the hierarchy.

Re: How I Made Google's "Web" View My Default Search

#113

Post by VikingCoder »

NayamAmarshe wrote: post #109
VikingCoder wrote: post #104

Earlier quoted context omitted.

Wikipedia shows an ad asking for money. So a hundred times a day, I open a new Wikipedia clone with no ads. I get perfect scores. Once I detect your bot has crawled it, I put ads all over it until the traffic finally goes away. No system is fool-proof.

But there's a difference between an embedded ad and a native ad. You can allow websites to just add an image or a section of an ad that does not hit a third-party endpoint. Native ads are not as intrusive and if they become intrusive, like a popup, you can penalize that. It sounds like a sound system.

This is a predator/prey relationship.

OP suggested they had a fool-proof way to get around that.

They're wrong.

Might there be some merit in their idea? Sure.

Is it so fool-proof that it would actually be worth the effort to build and maintain their own crawler and index?

I highly doubt it.

Re: How I Made Google's "Web" View My Default Search

#114

Post by immibis »

acatton wrote: post #63
lelanthran wrote: post #11

I want to make my own search engine, one day, with my own crawler. There is an SEO-proof way to determine what the ranking of a site should be - penalise it for each advertisement, penalise further for delivering different content to the crawler[1], allow logged-in users to down-rank a site, etc. Basically, a site starts off with a perfect score, then gets penalised for each violation, for each dark-pattern, for each…

I wish you luck and I hope you succeed, but you make it sound much much easier than what it would be. First of all, you're going to drown in hardware costs, if you run your own hardware. If you run on AWS, you will be the largest AWS customer. 2 years ago, when Google was still displaying result counts, I got 1.3 billions results for "sushi"[1]. This means that if you use a reverse index to lookup your results, the "…

Kagi obviously manages it. The Internet is big, but most of it is spam, which you can discard. You don't need relationships between all pages, just all websites. You should track the quality of the website, not each page. Google result counts are fake.

Re: How I Made Google's "Web" View My Default Search

#116

Post by Spivak »

tqwhite wrote: post #105

I solved this Google problem by paying someone else for search. I have switched to kagi.com, a so-called, "paid, ad-free search engine". I have not yet found one drawback to using it. Unlike previous attempts to switch from Google, I have not ever felt like I was getting substandard answers or thought of switching back. It's been months. And, it is lovely. The lack of ads and of it having no manipulative motivation m…

Same, it's crazy how easy of a W paying for Kagi is. For a lot of the privacy focused "Google alternatives" you're typically getting privacy in exchange for worse results but on Kagi it's the opposite. They have the !g bang like DuckDuckGo but unlike DDG I've never reached for it. I learned really quick that if Kagi had bad results that Google had even worse ones.

Re: How I Made Google's "Web" View My Default Search

#119

Post by WarOnPrivacy »

wernsey wrote: post #18

I had to do some digging on SuperUser [0] to get this working in Firefox, because it has the "Add Search Engine" button disabled by default. * Go to `about:config` * Change `browser.urlbar.update2.engineAliasRefresh` to true * Now you can go to `about:preferences#search` * ...and click the "Add" button below Search Shortcuts * Add ` https://www.google.com/search?q=%s&udm=14 ` as your URL. * Set the name to "Google We…

> Change `browser.urlbar.update2.engineAliasRefresh` to true

Just a note that this flag didn't exist in my Ffx 127.0b4. Created to True = surfaced the Add button as advertised.

Re: How I Made Google's "Web" View My Default Search

#120

Post by nicce »

chongli wrote: post #98
nicce wrote: post #85

Earlier quoted context omitted.

Yeah, there is a dilemma. To get a wider audience, you need very good search results. But if you search quality depends on the amount of users, how do you get in there? People like power, that keeps them moderating (looking at Facebook groups). What kind of reward system works here?

Well the baseline search quality should not rely on users. It should be great out of the box until spammers begin targeting it directly. Relying on users to moderate should happen organically as the number of users grow, which ideally would be timed to offset the growth in spam targeted at the engine.

The issue is that it needs the really really good on baseline before people start switching from other services.

But if you are able provide better baseline than others, then you might not even need the users to be "better. With users, you would be then exceptional.

Post reply on HN