Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I Made Google's "Web" View My Default Search

tedium.co

Post reply on HN
Print view View original

131–140 of 154 posts

Re: How I Made Google's "Web" View My Default Search

#131

Post by tambourine_man »

lelanthran wrote: post #11

I want to make my own search engine, one day, with my own crawler. There is an SEO-proof way to determine what the ranking of a site should be - penalise it for each advertisement, penalise further for delivering different content to the crawler[1], allow logged-in users to down-rank a site, etc. Basically, a site starts off with a perfect score, then gets penalised for each violation, for each dark-pattern, for each…

I've long felt that if we separated the problem we could open the flood gates for a lot of innovation. Crawling should be its own thing. That is much easily distributed and decentralized.

We could then freely experiment with algorithms and strategies on that open database.

Re: How I Made Google's "Web" View My Default Search

#132

Post by WarOnPrivacy »

mark_l_watson wrote: post #122
tqwhite wrote: post #105

I solved this Google problem by paying someone else for search. I have switched to kagi.com, a so-called, "paid, ad-free search engine". I have not yet found one drawback to using it. Unlike previous attempts to switch from Google, I have not ever felt like I was getting substandard answers or thought of switching back. It's been months. And, it is lovely. The lack of ads and of it having no manipulative motivation m…

I tried Kagi for a while and found it to be excellent. I settled on Perplexity for my own use however. I still use Duck Duck Go, Google, and Bing-ChatGPT. Not sure why, but sometimes I like variety.

> I tried Kagi for a while and found it to be excellent. I settled on Perplexity for my own use however.

Is Perplexity a page-results search engine?

Re: How I Made Google's "Web" View My Default Search

#133

Post by chongli »

theturtletalks wrote: post #95
chongli wrote: post #93

Earlier quoted context omitted.

There’s quite a lot of dark patterns to watch out for. One is including text on a page meant to be read by your crawler but which is hidden from users by various CSS tricks. To get around this you’d either need to specifically blacklist certain CSS / DOM structures or do something very radical: render the page and then attempt to OCR it back into text to match against the original text.

Remember when Google said they would penalize websites that use dark patterns like Quora, Pinterest, Twitter, etc? Google can easily penalize those websites for their dark patterns so websites and blogs by actual people rise to the top but they don’t.

I think they actually do that but then specifically white-list those sites. It’s really annoying!

Re: How I Made Google's "Web" View My Default Search

#134

Post by acatton »

immibis wrote: post #114
acatton wrote: post #63

Earlier quoted context omitted.

I wish you luck and I hope you succeed, but you make it sound much much easier than what it would be. First of all, you're going to drown in hardware costs, if you run your own hardware. If you run on AWS, you will be the largest AWS customer. 2 years ago, when Google was still displaying result counts, I got 1.3 billions results for "sushi"[1]. This means that if you use a reverse index to lookup your results, the "…

Kagi obviously manages it. The Internet is big, but most of it is spam, which you can discard. You don't need relationships between all pages, just all websites. You should track the quality of the website, not each page. Google result counts are fake.

I don't know how Kagi manages it. I suspect like duckduckgo, they index a little bit on their own, and use a GBY [1] as a back up. According to Seirdy[1], they use Brave in the background. Brave just burn their cryptomoney to build a search engine. Don't get me wrong, I like what their doing, but it was easy for them to start since they bootstraped from Cliqz' index[2]

And for the second part, you do need to store the relationship between keywords and pages, that's what I was talking about. You cannot store a relationship between "types of water" and "reddit.com" you need to store it between "type of water" and "reddit.com/r/hydrohomies/..."

[1] https://seirdy.one/posts/2021/03/10/search-engines-with-own-...

[2] https://brave.com/blog/brave-search/

Re: How I Made Google's "Web" View My Default Search

#135

Post by acatton »

lelanthran wrote: post #94
acatton wrote: post #63

Earlier quoted context omitted.

I wish you luck and I hope you succeed, but you make it sound much much easier than what it would be. First of all, you're going to drown in hardware costs, if you run your own hardware. If you run on AWS, you will be the largest AWS customer. 2 years ago, when Google was still displaying result counts, I got 1.3 billions results for "sushi"[1]. This means that if you use a reverse index to lookup your results, the "…

I was just idly threatening to do something, not actually starting a venture to do it. But, in any case, lets go with your numbers for running costs: in 2024 money, what is your estimation of the running costs? I ask because there have been a number of new search engines pop up, and they have nowhere near the expenditure power of google, yet they still have devoted followings. > The second challenge is your ranking.…

Most search engine popping up are just a rebranding of bing and yandex. There is a good article about it: https://seirdy.one/posts/2021/03/10/search-engines-with-own-...

Regarding your question of cost, if you buy the cheapest hardware possible, and manage it yourself, because you don't want to pay AWS' premium, for storage (again, assuming to discard 90% of the pages, and only do english) you'll be at least at €80k (12 × €3000 disk servers with 300 × €150 HDDs) of upfront cost, then, if you colocate at Hetzner, just for storage, you'll bet at €500/month + ~€3k of electricity that you need to pay yourself.

My gut feeling is that this would cost (only 10% of the English-speaking web) ~€150k of upfront cost and €5k/month to run. Assuming you buy the cheapest everything, and do your own admin sys. And this is not forecasting growth, serving ads, etc...

Re: How I Made Google's "Web" View My Default Search

#136

Post by Aachen »

hrnnnnnn wrote: post #57
triyambakam wrote: post #4

Honest question - why still use Google at all? Whenever DDG became good enough (seven? ten? years ago?) I used it exclusively. Lately I use a mix and have moved on from DDG, but I still never went back to Google and don't understand how people can tolerate it; I find the results really bad.

For many of my searches DDG seems to completely ignore one or more of my keywords, usually giving me something more popular but less relevant. Almost every time when I try the same search on google it works.

Keep in mind that if you take only a specific type of searches to Google, in this case the ones DDG struggles with, it may just be that Google is good at that class specifically (like natural language or a query doing well with long literal string matching or something) and that makes it appear more competent while others, using Google all the time, are annoyed by it a lot of the time

Re: How I Made Google's "Web" View My Default Search

#137

Post by nine_k »

nicce wrote: post #120
chongli wrote: post #98

Earlier quoted context omitted.

Well the baseline search quality should not rely on users. It should be great out of the box until spammers begin targeting it directly. Relying on users to moderate should happen organically as the number of users grow, which ideally would be timed to offset the growth in spam targeted at the engine.

The issue is that it needs the really really good on baseline before people start switching from other services. But if you are able provide better baseline than others, then you might not even need the users to be "better. With users, you would be then exceptional.

I suppose services like Gmail benefit a lot from users who mark the occasional spam letter in their inboxes. While each such marking is inconsequential, correlated over many users, they allow to detect new and unusual waves of spam, and to start filtering them for everyone.

A similar mechanism could work for a search engine. It would be harder to game if it required high karma (many other users matching your judgment before it starts to count), and a paid account, like with Kagi.

Re: How I Made Google's "Web" View My Default Search

#138

Post by p3rls »

lelanthran wrote: post #11

I want to make my own search engine, one day, with my own crawler. There is an SEO-proof way to determine what the ranking of a site should be - penalise it for each advertisement, penalise further for delivering different content to the crawler[1], allow logged-in users to down-rank a site, etc. Basically, a site starts off with a perfect score, then gets penalised for each violation, for each dark-pattern, for each…

Counterpoint: I'm spending thousands on stuff like APIs to graph out data for my users and would like to be compensated.

The existing system could work with tweaks. I like metrics like time-on-page, load times, and "domain authority". With the right weights on the algorithm, real practical guidelines and a robust system of manual actions (hopefully as transparent as possible) I believe something like Google can last for as long as the internet does.

Re: How I Made Google's "Web" View My Default Search

#139

Post by theturtletalks »

chongli wrote: post #133
theturtletalks wrote: post #95

Earlier quoted context omitted.

Remember when Google said they would penalize websites that use dark patterns like Quora, Pinterest, Twitter, etc? Google can easily penalize those websites for their dark patterns so websites and blogs by actual people rise to the top but they don’t.

I think they actually do that but then specifically white-list those sites. It’s really annoying!

That’s even more damning. So the established sites are playing by different SEO rules altogether?

Re: How I Made Google's "Web" View My Default Search

#140

Post by chongli »

theturtletalks wrote: post #139
chongli wrote: post #133

Earlier quoted context omitted.

I think they actually do that but then specifically white-list those sites. It’s really annoying!

That’s even more damning. So the established sites are playing by different SEO rules altogether?

I don’t have any proof. I’m just taking it as a given that this is what they’re doing because those sites are still engaging in cloaking despite Google making it clear they have a policy against cloaking.
Post reply on HN