Live data from Hacker News

Creating a search engine for fun and because Google sucks

vincents.dev

21–30 of 45 posts

Re: Creating a search engine for fun and because Google sucks

#21
post #6
post #3

Google is a fine search engine.

Google is fine unless you need something really specific, chatgpt is better at this point for a lot of queries because Google give you the most simple answer no matter how specific you got with your keywords. I still miss the ability to use "".

[deleted]

Re: Creating a search engine for fun and because Google sucks

#23
post #10
post #5

So your "search engine" uses Google and DuckDuckGo to get results? and you are creating it because "Google sucks"? Am I missing something here?

Kagi does that, is a paid subscription and HN does praise it a lot.

Kagi implements search from scratch, from the indexing through querying. This looks more like a frontend makeover

Re: Creating a search engine for fun and because Google sucks

#24
post #4
post #3

Google is a fine search engine.

It's fashionable to bash Google and there's probably good reasons to do so. But I agree, their search is still good (enough) for me. Over the years, I've tried others, including DDG, but ultimately returned to Google every time. So, until a better one comes along, Google's search is fine for me.

Google is mostly fine if you know what you are looking for and if it's not an SEO / ad target topic.

If those two don't apply Google is often pretty much useless, just giving you low quality/ AI spam blog posts, useless ad-ridden product comparison sites and marketing pages.

There's a reason why adding "reddit" to Google searches has almost become a meme now.

Re: Creating a search engine for fun and because Google sucks

#26
post #9

I have been thinking about this. I’d like to index the blogosphere and some knowledge base sites and stuff and exclude most of the Top1000 websites. I imagine I could do it on consumer hardware for less than $10-20k. Perhaps common crawl has done much of the heavy lifting already and I just have an indexing task.

Robust crawling with all the gratuitous anti-bot infra out there (requesting the rss feed that is specifically meant to be consumed by machine for a public gov site is 'protected' by Cloudflare's anti-botting. Seriously?) takes way more work than you think.

Re: Creating a search engine for fun and because Google sucks

#27
post #4

Earlier quoted context omitted.

It's fashionable to bash Google and there's probably good reasons to do so. But I agree, their search is still good (enough) for me. Over the years, I've tried others, including DDG, but ultimately returned to Google every time. So, until a better one comes along, Google's search is fine for me.

Google is mostly fine if you know what you are looking for and if it's not an SEO / ad target topic. If those two don't apply Google is often pretty much useless, just giving you low quality/ AI spam blog posts, useless ad-ridden product comparison sites and marketing pages. There's a reason why adding "reddit" to Google searches has almost become a meme now.

Show me an actual query then.

Why do people always jump on 'umm Google bad' trends? It's all well and good to say it, but as someone who hasn't ever experienced this (apparently common) problem before, I would like to see to believe.

Re: Creating a search engine for fun and because Google sucks

#28
post #17

Creating a thin, non-working wrapper around DuckDuckGo as a "hire me" ad, claiming it's a search engine and mentioning Google in the headline because it'll get more clicks.

Yes, I don’t know how that reached the frontpage. A failed 15 minute experiment from someone who has no clue what they are doing

Re: Creating a search engine for fun and because Google sucks

#29
As someone who cares about their online searches actually being good, fast and private, I cannot recommend SearXNG more. https://github.com/searxng/searxng/

It's a metasearch engine that can query multiple search providers at once, including google, so you're not missing out on the good results you expect. Pick an instance at https://searx.space/ and tell your friends!

Re: Creating a search engine for fun and because Google sucks

#30
post #9

I have been thinking about this. I’d like to index the blogosphere and some knowledge base sites and stuff and exclude most of the Top1000 websites. I imagine I could do it on consumer hardware for less than $10-20k. Perhaps common crawl has done much of the heavy lifting already and I just have an indexing task.

Robust crawling with all the gratuitous anti-bot infra out there (requesting the rss feed that is specifically meant to be consumed by machine for a public gov site is 'protected' by Cloudflare's anti-botting. Seriously?) takes way more work than you think.

The way to get around anti-bot stuff as a search engine crawler is to be upfront about what you are doing, as well as to respect robots.txt, and rate limits, and use a gratuitous crawl delay (1 sec is the bare minimum).

You can register with cloudflare as a search engine crawler and they largely will let your traffic through. If you get blocked by individual sites, you can usually just email them explaining the situation and get unblocked.

I'm having very little issues with this.

Post reply on HN