Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Updates to our web search products and Programmable Search Engine capabilities

programmablesearchengine.googleblog.com

Post reply on HN
Print view View original

31–40 of 207 posts

Re: Updates to our web search products and Programmable Search Engine capabilities

#31

Post by saltysalt »

orf wrote: post #22
saltysalt wrote: post #17

Earlier quoted context omitted.

I built my own web search index on bare metal, index now up to 34m docs: https://greppr.org/ People rely too much on other people's infra and services, which can be decommissioned anytime. The Google Graveyard is real.

Number of docs isn’t the limiting factor. I just searched for “stackoverflow” and the first result was this: https://www.perl.com/tags/stackoverflow/ The actual Stackoverflow site was ranked way down, below some weird twitter accounts.

I don't weight home pages in any way yet to bump them up, it's just raw search on keyword relevance.

Re: Updates to our web search products and Programmable Search Engine capabilities

#33

Post by johnofthesea »

tweetle_beetle wrote: post #18
vaylian wrote: post #8

Meanwhile in Europe: Qwant and Ecosia team up to build their own search index: https://blog.ecosia.org/eusp/

It's a noble effort, but they're so late to the game that it's hard to see them making a significant dent. I hope I'm wrong. They were: > aiming to serve 30% of French search queries [by end of 2025] https://blog.ecosia.org/launching-our-european-search-index/

Better late than never.

> The French index is at an advanced stage of completion, we have started creating the German language index, and the English one should start shortly. All progress is quickly integrated into the Qwant STAAN API.

https://noc.social/@327ppm/115934198650900394

Re: Updates to our web search products and Programmable Search Engine capabilities

#34

Post by jamesbelchamber »

Are competing search indexes (Bing, Ecosia/Qwant, etc) objectively worse in significant ways, or is Google just so entrenched that people don't want to "risk it" with another provider (and/or preferences and/or inertia).

I suppose I'm asking whether this is actually a _good thing_ in that it will stimulate competition in the space, or if it's just a case that Google's index is now too good for anyone to reasonably catch up at this point.

Re: Updates to our web search products and Programmable Search Engine capabilities

#35

Post by marginalia_nu »

bovermyer wrote: post #19

I'm curious about what it would take to build my own "toy" search engine with its own index. Anyone ever tried this?

Yeah that's where I started out in 2021. Been at it for almost 5 years now, last three of which full time. I'm indexing about 1.1 billion documents now off a single server.

Hard part is doing it at any sort of scale and producing useful results. It's easy to build something that indexes a few million documents. Pushing into billions is a bigger challenge, as you start needing a lot of increasingly intricate bespoke solutions.

Devlog here:

https://www.marginalia.nu/tags/search-engine/

And search engine itself:

https://marginalia-search.com/

(... though it operates a bit sub-optimally now as I'm using a ton of CPU cores to migrate the index to use postings lists compression, will take about 4-5 days I think).

Re: Updates to our web search products and Programmable Search Engine capabilities

#36

Post by marginalia_nu »

reddalo wrote: post #32
bovermyer wrote: post #19

I'm curious about what it would take to build my own "toy" search engine with its own index. Anyone ever tried this?

Good luck scraping websites without being blocked, if you're not Google.

Well you'll get blocked some places but it's not too big of a deal. If you're running an above board operation, you can surprisingly often successfully just email the admin explaining what you're doing, and ask to be unblocked.

Re: Updates to our web search products and Programmable Search Engine capabilities

#37

Post by 01jonny01 »

jamesbelchamber wrote: post #34

Are competing search indexes (Bing, Ecosia/Qwant, etc) objectively worse in significant ways, or is Google just so entrenched that people don't want to "risk it" with another provider (and/or preferences and/or inertia). I suppose I'm asking whether this is actually a _good thing_ in that it will stimulate competition in the space, or if it's just a case that Google's index is now too good for anyone to reasonably ca…

The beauty about Google Programmable Search across the entire web is that it's free and users can make money by linking it their Adsense account.

Bing charge per query for the average user. Ecosia and Qwant use Bing to power their results, probably under some type of license, which results in them paying much less per query than a normal user.

Re: Updates to our web search products and Programmable Search Engine capabilities

#38

Post by altairprime »

Gigachad wrote: post #26
tweetle_beetle wrote: post #18

Earlier quoted context omitted.

It's a noble effort, but they're so late to the game that it's hard to see them making a significant dent. I hope I'm wrong. They were: > aiming to serve 30% of French search queries [by end of 2025] https://blog.ecosia.org/launching-our-european-search-index/

I feel like soon there won’t even be a point having a search engine since almost the entire internet will be useless AI slop.

It's as though full-text search of websites you've never heard of was a mistake :)

PageRank wouldn't exist without webrings, directories, and forums you could only search individually, and we thrived on that Internet.

Welcome back, ye olde Internet.

Re: Updates to our web search products and Programmable Search Engine capabilities

#39

Post by thayne »

jamesbelchamber wrote: post #34

Are competing search indexes (Bing, Ecosia/Qwant, etc) objectively worse in significant ways, or is Google just so entrenched that people don't want to "risk it" with another provider (and/or preferences and/or inertia). I suppose I'm asking whether this is actually a _good thing_ in that it will stimulate competition in the space, or if it's just a case that Google's index is now too good for anyone to reasonably ca…

Bing recently shut down their API product, which was already very expensive.

If you want programmatic access to search results there aren't really many options left.

Re: Updates to our web search products and Programmable Search Engine capabilities

#40

Post by embedding-shape »

marginalia_nu wrote: post #13
doublerabbit wrote: post #9

Earlier quoted context omitted.

Kagi

They published this the other day: https://blog.kagi.com/waiting-dawn-search Which saw some discussion on HN.

> some discussion

~450 score, ~247 comments and still on /best ("Most-upvoted stories of the last 48 hours"):

https://news.ycombinator.com/item?id=46708678 - "Waiting for dawn in search: Search index, Google rulings and impact on Kagi"

Post reply on HN