Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Updates to our web search products and Programmable Search Engine capabilities

programmablesearchengine.googleblog.com

Post reply on HN
Print view View original

191–200 of 207 posts

Re: Updates to our web search products and Programmable Search Engine capabilities

#191

Post by bicepjai »

Why don’t we have something more “torrent-like” for search?

Imagine a decentralized network where volunteers run crawler nodes that each fetch and extract a tiny slice of the web. Those partial results get merged into open, versioned indexes that can be distributed via P2P (or mirrored anywhere). Then anyone can build ranking, vertical search, or specialized tools on top of that shared index layer.

I get that reproducing Google’s “Coca-Cola formula” (ranking, spam fighting, infra, freshness, etc.) is probably unrealistic. But I’d happily use the coconut-water version: an open baseline index that’s good enough, extensible, and not owned by a single gatekeeper.

I know we have common crawl, but small processing nodes can be more efficient and fresh

Re: Updates to our web search products and Programmable Search Engine capabilities

#192

Post by pona-a »

bicepjai wrote: post #191

Why don’t we have something more “torrent-like” for search? Imagine a decentralized network where volunteers run crawler nodes that each fetch and extract a tiny slice of the web. Those partial results get merged into open, versioned indexes that can be distributed via P2P (or mirrored anywhere). Then anyone can build ranking, vertical search, or specialized tools on top of that shared index layer. I get that reprodu…

Look up YaCy. This might be close to what you imagine

Re: Updates to our web search products and Programmable Search Engine capabilities

#193

Post by bicepjai »

pona-a wrote: post #192
bicepjai wrote: post #191

Why don’t we have something more “torrent-like” for search? Imagine a decentralized network where volunteers run crawler nodes that each fetch and extract a tiny slice of the web. Those partial results get merged into open, versioned indexes that can be distributed via P2P (or mirrored anywhere). Then anyone can build ranking, vertical search, or specialized tools on top of that shared index layer. I get that reprodu…

Look up YaCy. This might be close to what you imagine

Thanks for that info, they are doing exactly what I was saying. Why is that not adopted widely ? Found HN posts

YaCy, a distributed Web Search Engine, based on a peer-to-peer network https://news.ycombinator.com/item?id=39612950

Re: Updates to our web search products and Programmable Search Engine capabilities

#194

Post by jsnell »

ipaddr wrote: post #155
jsnell wrote: post #94

Earlier quoted context omitted.

They then go on to say that they pay a 3rd party company to scrape Google results (and serve those scraped results to their users). So their search engine is indeed based on unauthorized and uncompensated use of Google's index. But since they're not using/paying for a supported API but just taking what they want, they indeed are unlikely to be impacted by this API turndown.

They get results from another provider who has authorized access. Google doesn't provide search results to unauthorized requests as many on tor have experienced.

No. They pay SerpApi to scrape Google. And SerpApi is currently being sued by Google for unauthorized scraping.

Kagi did make comments for years implying that they had a deal with Google for search results, but their latest blog post makes it clear that is not true and was never true.

Re: Updates to our web search products and Programmable Search Engine capabilities

#195

Post by danans »

snowwrestler wrote: post #110
saltysalt wrote: post #85

Earlier quoted context omitted.

PageRank was an innovative idea in the early days of the Internet when trust was high, but yes it's absolutely gamed now and I would be surprised if Google still relies on it. Fair play to them though, it enabled them to build a massive business.

Google’s biggest search signal now is aggregate behavioral data reported from Chrome. That pervasive behavioral surveillance is the main reason Apple has never allowed a native Chrome app on iOS. It’s also why it is so hard to compete with Google. You guys are talking about techniques for analyzing the corpus of the search index. Google does that and has a direct view into how millions of people interact with it.

> That pervasive behavioral surveillance is the main reason Apple has never allowed a native Chrome app on iOS.

There is a native Chrome app on iOS. It gets all the same url visit data as Chrome on other platforms.

Apple blocks 3rd party renderers and JS engines on iOS to protect its App Store from competition that might deliver software and content through other channels that they don't take a cut of.

Re: Updates to our web search products and Programmable Search Engine capabilities

#196

Post by senko »

consumer451 wrote: post #111

Dumb question: I keep seeing posts about how ~"the volume of AI scrapers is making hosting untenable." There must a ton of new full-web datasets out there, right? What are the major hurdles that prevent the owners of these datasets from providing them to third parties via API? Is it the quality of SERP, or staleness? Otherwise, this seems like a potentially lucrative pivot/side hustle?

> There must a ton of new full-web datasets out there, right?

Sadly, no. There's CommonCrawl (https://commoncrawl.org/) which still, sadly, far removed from "full-web dataset."

So everyone runs their own search instead, hammering the sites, going into gray areas (you either ignore robots.txt or your results suck), etc. It's a tragedy of the commons that keeps Google entrenched: https://senkorasic.com/articles/ai-scraper-tragedy-commons

Re: Updates to our web search products and Programmable Search Engine capabilities

#197

Post by vagab0nd »

01jonny01 wrote: post #2

Google quietly announced that Programmable Search (ex-Custom Search) won’t allow new engines to “search the entire web” anymore. New engines are capped at searching up to 50 domains, and existing full-web engines have until Jan 1, 2027 to transition. If you actually need whole-web search, Google now points you to an “interest form” for enterprise solutions (Vertex AI Search etc.), with no public pricing and no guaran…

Damn, I just wrote a note "search is free" in my aggressively-automate-everything-using-llms personal project plan.md. I guess not anymore.

Re: Updates to our web search products and Programmable Search Engine capabilities

#198

Post by saltysalt »

1718627440 wrote: post #184
saltysalt wrote: post #159

Earlier quoted context omitted.

It does not use WordPress.

I'm sorry, I am dumb and visited http://grepper.org/ . Where does your name come from I guess from grep for the WWW?

Yes correct, that is where the name comes from.

Re: Updates to our web search products and Programmable Search Engine capabilities

#199

Post by saltysalt »

1718627440 wrote: post #186
saltysalt wrote: post #182

Earlier quoted context omitted.

It's a difficult problem to fix, you can set an Accept-Language header on crawl requests but his only works if the target website uses "Content Negotiation." Some sites ignore headers and determine language based on the IP address (Geo-IP) or the URL structure (e.g., /es/ vs /en/), basically a mess...

I don't get the problem you claim. You crawl something and get a document in whatever language the site delivers you. You know the language of that document with the lang=... attribute of the document. What results you show for a given language is under your control and not influenced by what the crawled site chose to serve to the crawler.

I'm working on the language improvements presently, but I need to clean out a lot of bad entries in my index. In essence what I am trying to say is many servers ignore "Accept-Language" so you have to rely on other means of detecting the language of the page reliably, e.g. inspecting the body content of the response. It's a non-trivial problem online.

Re: Updates to our web search products and Programmable Search Engine capabilities

#200

Post by saltysalt »

bflesch wrote: post #173
saltysalt wrote: post #17

Earlier quoted context omitted.

I built my own web search index on bare metal, index now up to 34m docs: https://greppr.org/ People rely too much on other people's infra and services, which can be decommissioned anytime. The Google Graveyard is real.

Thanks for sharing, this is really impressive. Can you talk a bit about your stack? The about page mentions grep but I'd assume it's a bit more complex than having a large volume and running grep over it ;) Is it some sort of custom database or did you keep it simple? Do you also run a crawler?

I huge Lucene index for storage and search, with a custom crawler that I wrote myself. It's a fun engineering problem.
Post reply on HN