Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Updates to our web search products and Programmable Search Engine capabilities

programmablesearchengine.googleblog.com

Post reply on HN
Print view View original

51–60 of 207 posts

Re: Updates to our web search products and Programmable Search Engine capabilities

#51

Post by johnofthesea »

saltysalt wrote: post #17
01jonny01 wrote: post #2

Google quietly announced that Programmable Search (ex-Custom Search) won’t allow new engines to “search the entire web” anymore. New engines are capped at searching up to 50 domains, and existing full-web engines have until Jan 1, 2027 to transition. If you actually need whole-web search, Google now points you to an “interest form” for enterprise solutions (Vertex AI Search etc.), with no public pricing and no guaran…

I built my own web search index on bare metal, index now up to 34m docs: https://greppr.org/ People rely too much on other people's infra and services, which can be decommissioned anytime. The Google Graveyard is real.

I tested it using a local keyword, as I normally do, and it took me to a Wikipedia page I didn’t know existed. So thanks for that.

Re: Updates to our web search products and Programmable Search Engine capabilities

#52

Post by SirHumphrey »

jamesbelchamber wrote: post #34

Are competing search indexes (Bing, Ecosia/Qwant, etc) objectively worse in significant ways, or is Google just so entrenched that people don't want to "risk it" with another provider (and/or preferences and/or inertia). I suppose I'm asking whether this is actually a _good thing_ in that it will stimulate competition in the space, or if it's just a case that Google's index is now too good for anyone to reasonably ca…

I can manage fine with other search indexes for English language searches; weather that is because others got better or google got worse i cannot tell, though I suspect the latter.

But for searching in more niche languages google is usually the only decent option and I have little hope that others will ever reach the scale where they could compete.

Re: Updates to our web search products and Programmable Search Engine capabilities

#53

Post by marginalia_nu »

echelon wrote: post #28
doublerabbit wrote: post #9

Earlier quoted context omitted.

Kagi

No wonder Kagi is angry. Google is a monopoly across several broad categories. They're also a taxation enterprise. Google Search took over as the URL bar for 91% of all web users across all devices. Since this intercepts trademarks and brand names, Google gets to tax all businesses unfairly. Tell your legislators in the US and the EU that Google shouldn't be able to sell ads against registered trademarks (+/- some ed…

Could probably argued that search access is an essential facility[1], though it doesn't appear antitrust law has anywhere near the same sort of enforcement it did in the past.

[1] https://en.wikipedia.org/wiki/Essential_facilities_doctrine

Re: Updates to our web search products and Programmable Search Engine capabilities

#54

Post by saltysalt »

orf wrote: post #50
saltysalt wrote: post #31

Earlier quoted context omitted.

I don't weight home pages in any way yet to bump them up, it's just raw search on keyword relevance.

Sure, but the point is results are not relevant at all? It’s cool though, and really fast

I'll work on that adjustment, it's fair feedback thanks!

Re: Updates to our web search products and Programmable Search Engine capabilities

#55

Post by saltysalt »

johnofthesea wrote: post #51
saltysalt wrote: post #17

Earlier quoted context omitted.

I built my own web search index on bare metal, index now up to 34m docs: https://greppr.org/ People rely too much on other people's infra and services, which can be decommissioned anytime. The Google Graveyard is real.

I tested it using a local keyword, as I normally do, and it took me to a Wikipedia page I didn’t know existed. So thanks for that.

It will throw up weird and interesting results sometimes ;-)

Re: Updates to our web search products and Programmable Search Engine capabilities

#56

Post by marginalia_nu »

rickette wrote: post #46
marginalia_nu wrote: post #35

Earlier quoted context omitted.

Yeah that's where I started out in 2021. Been at it for almost 5 years now, last three of which full time. I'm indexing about 1.1 billion documents now off a single server. Hard part is doing it at any sort of scale and producing useful results. It's easy to build something that indexes a few million documents. Pushing into billions is a bigger challenge, as you start needing a lot of increasingly intricate bespoke s…

Curious on what (how much) hardware your running this.

Currently running off

AMD EPYC 7543 x2 for 64 cores/128 threads

512 GB RAM

~ 90 TB of PM9A3 SSDs across 12 physical devices

Storage is not very full though. I'm probably using about a third of it at this point.

Re: Updates to our web search products and Programmable Search Engine capabilities

#57

Post by halapro »

sreekanth850 wrote: post #42

Never build a product with core feature depending on a third-party, you will eventually get fucked up for sure. always have a 70:30 rule for revenue where 70% is core independent features.

Soon you'll find that you cannot exist on the web without relying on third parties. Sometimes you'll even have trouble getting paid thanks to the painful existence of payment processors.

Re: Updates to our web search products and Programmable Search Engine capabilities

#58

Post by nemosaltat »

doublerabbit wrote: post #9
throwaway_20357 wrote: post #4

Earlier quoted context omitted.

What are some of the niche search engines build on Google's index affected by this?

Kagi

> Kagi This seems to be true, but more indirectly. From Kagi’s blog [0] which is a follow up to a Kagi blog post from last year [1].

[0]> Google: Google does not offer a public search API. The only available path is an ad-syndication bundle with no changes to result presentation - the model Startpage uses. Ad syndication is a non-starter for Kagi’s ad-free subscription model.[^1]

[0]> The current interim approach (current as of Jan 21, 2026)

[0]> Because direct licensing isn’t available to us on compatible terms, we - like many others - use third-party API providers for SERP-style results (SERP meaning search engine results page). These providers serve major enterprises (according to their websites) including Nvidia, Adobe, Samsung, Stanford, DeepMind, Uber, and the United Nations.

I’m an avid Kagi user, and it seems like Kagi and some other notable interested parties have _already_ been unable to do get what they want/need with Google’s index.

[0]> The fact that we - and companies like Stanford, Nvidia, Adobe, and the United Nations - have had to rely on third-party vendors is a symptom of the closed ecosystem, not a preference.

Hopefully someone here can clarify for me, or enumerate some of these “third-party vendors” who seem like they will/might/could be directly affected by this.

[0] antibabelic > relevant https://blog.kagi.com/waiting-dawn-search [1] https://blog.kagi.com/dawn-new-era-search > [^1]: A note on Google’s existing APIs: Google offers PSE, designed for adding search boxes to websites. It can return web results, but with reduced scope and terms tailored for that narrow use case. More recently, Google offers Grounding with Google Search through Vertex AI, intended for grounding LLM responses. Neither is general-purpose index access. Programmable Search Engine is not designed for building competitive search. Grounding with Google Search is priced at $35 per 1,000 requests - economically unviable for search at scale, and structured as an AI add-on rather than standalone index syndication. These are not the FRAND terms the market needs

Re: Updates to our web search products and Programmable Search Engine capabilities

#60

Post by 01jonny01 »

cubefox wrote: post #43

Is this perhaps to prevent ChatGPT, Claude and Grok to use Google Search? It would make sense for Google to keep that ability for Gemini.

I suspect its going to hurt the indie developers and small start-ups who do not have special licensing agreements.
Post reply on HN