Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Updates to our web search products and Programmable Search Engine capabilities

programmablesearchengine.googleblog.com

Post reply on HN
Print view View original

171–180 of 207 posts

Re: Updates to our web search products and Programmable Search Engine capabilities

#171

Post by agosta »

mrweasel wrote: post #16
Antibabelic wrote: post #3

Relevant: Waiting for dawn in search: Search index, Google rulings and impact on Kagi https://news.ycombinator.com/item?id=46708678

This might be me reading it wrong, but isn't shutting down the full-web search going against the ruling mentioned in the Kagi post? > Google must provide Web Search Index data (URLs, crawl metadata, spam scores) at marginal cost. Maybe they're shutting down the good integration and then Kagi, Ecosia and others can buy index data in an inconvenient way going forward?

Among the various rulings, Google is supposed to provide access at market rates... which they are. At least for what is published: $5 per thousand queries is market rate for a product like this - see Brave's Search API pricing https://api-dashboard.search.brave.com/app/plans?tab=ai.

Granted, that is scoped to 50 domains. But we don't know if the enterprise package, which allows full web search, isn't roughly market rate.

Re: Updates to our web search products and Programmable Search Engine capabilities

#172

Post by whatarethembits »

blell wrote: post #65
johnofthesea wrote: post #33

Earlier quoted context omitted.

Better late than never. > The French index is at an advanced stage of completion, we have started creating the German language index, and the English one should start shortly. All progress is quickly integrated into the Qwant STAAN API. https://noc.social/@327ppm/115934198650900394

They can build whatever they want with lots of #hashtags and public money, but that doesn't mean they'll get 30% of French people to use it. But of course they managed to cut themselves a nice salary with EU funds, paid in part by me and you, so that's all that matters.

The landscape has completely shifted now; now more than ever, there's a real need for Europe to pivot from relying too much on an adversary.

Re: Updates to our web search products and Programmable Search Engine capabilities

#173

Post by bflesch »

saltysalt wrote: post #17
01jonny01 wrote: post #2

Google quietly announced that Programmable Search (ex-Custom Search) won’t allow new engines to “search the entire web” anymore. New engines are capped at searching up to 50 domains, and existing full-web engines have until Jan 1, 2027 to transition. If you actually need whole-web search, Google now points you to an “interest form” for enterprise solutions (Vertex AI Search etc.), with no public pricing and no guaran…

I built my own web search index on bare metal, index now up to 34m docs: https://greppr.org/ People rely too much on other people's infra and services, which can be decommissioned anytime. The Google Graveyard is real.

Thanks for sharing, this is really impressive.

Can you talk a bit about your stack? The about page mentions grep but I'd assume it's a bit more complex than having a large volume and running grep over it ;)

Is it some sort of custom database or did you keep it simple? Do you also run a crawler?

Re: Updates to our web search products and Programmable Search Engine capabilities

#174

Post by sumtechguy »

embedding-shape wrote: post #146
zitterbewegung wrote: post #139

Earlier quoted context omitted.

I know that duckduckgo uses Microsoft Bing Custom search and honestly it is a much more robust system since you don't have to worry about Google axing it. https://www.customsearch.ai

Instead you worry about Microsoft axing it? Sure, it might take 3 years instead of 6 months, and the shutdown period would be 1 year instead of 1 month, but hardly either are long-term solutions.

Having been downsizing my horde of computer junk I have several large boxes of full MSDN disc sets. There is a graveyard of MS stuff that is no longer supported. The only thing with MS is they seem to give you a better off ramp usually than 'oh well sucks to be you'.

Re: Updates to our web search products and Programmable Search Engine capabilities

#175

Post by jpalepu33 »

This is a clear example of why building on proprietary APIs is risky for indie devs and small startups. I've seen similar patterns with Twitter's API restrictions and other platforms gradually closing down their ecosystems.

For anyone affected: consider this a forcing function to either: 1. Build your own lightweight search infrastructure (tools like Meilisearch, Typesense make this more accessible now) 2. Use adversarial interop via services like SerpAPI (though Google is already taking legal action there) 3. Pivot to specialized vertical search where you control the data sources

The real lesson here is the importance of owning your core value proposition. If your product's moat depends entirely on a third-party API that can be yanked away with 12 months notice, you don't really have a sustainable business.

Google is essentially saying: indie search is dead, pay enterprise prices or leave. This will probably accelerate the trend toward specialized, domain-specific search engines that don't rely on Google's index at all.

Re: Updates to our web search products and Programmable Search Engine capabilities

#176

Post by marginalia_nu »

riku_iki wrote: post #170
marginalia_nu wrote: post #35

Earlier quoted context omitted.

Yeah that's where I started out in 2021. Been at it for almost 5 years now, last three of which full time. I'm indexing about 1.1 billion documents now off a single server. Hard part is doing it at any sort of scale and producing useful results. It's easy to build something that indexes a few million documents. Pushing into billions is a bigger challenge, as you start needing a lot of increasingly intricate bespoke s…

I assume you able to monetize it since you work on it full time?

I'm mostly living off grants and donations at this point, but the plan down the line is to polish it up well enough to make some money off providing an API like the one Google is making it a hassle to access with this change :-)

Re: Updates to our web search products and Programmable Search Engine capabilities

#177

Post by joelboersma »

I've been occasionally working on a toy project that's basically "Google search in a TUI" that used this API. I was already planning on adding Brave Search as an option for a different backend, and I was heavily considering making it the default just because it's much easier to set up on the user's end. This is the straw that broke the camel's back.

Re: Updates to our web search products and Programmable Search Engine capabilities

#178

Post by [deleted] »

embedding-shape wrote: post #164
aylmao wrote: post #160

Earlier quoted context omitted.

> it might take 3 years instead of 6 months, and the shutdown period would be 1 year instead of 1 month This matters much more than people (and evidently those within Google) realize

It matters more the shorter your future planning is. Neither works if you're looking forward 3-4 years, for example.

[deleted]

Re: Updates to our web search products and Programmable Search Engine capabilities

#179

Post by consumer451 »

Tenemo wrote: post #166
Terretta wrote: post #152

Earlier quoted context omitted.

> the volume of AI scrapers is making hosting untenable Aside from that potential, it's also not true. A Pentium Pro or PIII SSE with circa 1998-99 Apache happily delivers a billion hits a month w/o breaking a sweat unless you think generating pages for every visit is better than generating pages when they change.

I think it is true that it is a real problem (EDIT: but doesn't necessarily make "hosting untenable"), but you are correct to point out that modern pages tend to be horribly optimized (and that's the source of the problem). Even "dynamic" pages using React/Next.js etc. could be pre-rendered and/or cached and/or distributed via CDNs. A simple cache or a CDN should be enough to handle pretty much any scrapping traffic…

OK, I agree with both of you. I am an old who is aware of NGINX and C10k. However, my question is: what are the economic or technical difficulties that prevent one of these new web-scale crawlers from releasing og-pagerank-api.com? We all love to complain about modern Google SERP, but what actually prevents that original Google experience from happening, in 2026? Is it not possible?

Or, is that what orgs like Perplexity are doing, but with an LLM API? Meaning that they have their own indexes, but the original q= SERP API concept is a dead end in the market?

Tone: I am asking genuine questions here, not trying to be snarky.

Re: Updates to our web search products and Programmable Search Engine capabilities

#180

Post by direwolf20 »

ipaddr wrote: post #155
jsnell wrote: post #94

Earlier quoted context omitted.

They then go on to say that they pay a 3rd party company to scrape Google results (and serve those scraped results to their users). So their search engine is indeed based on unauthorized and uncompensated use of Google's index. But since they're not using/paying for a supported API but just taking what they want, they indeed are unlikely to be impacted by this API turndown.

They get results from another provider who has authorized access. Google doesn't provide search results to unauthorized requests as many on tor have experienced.

Residential proxies are also cheaper than you might realize.
Post reply on HN