Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Updates to our web search products and Programmable Search Engine capabilities

programmablesearchengine.googleblog.com

Post reply on HN
Print view View original

181–190 of 207 posts

Re: Updates to our web search products and Programmable Search Engine capabilities

#181

Post by direwolf20 »

ColinHayhurst wrote: post #78
solarkraft wrote: post #45

This will significantly impact (quite possibly kill) Startpage and Ecosia, who are effectively white-label Google, right? What alternatives are there besides Bing? Is it really so hard that it’s not considered worth doing? Some of the AI companies (Perplexity, Anthropic) seem to have managed to get their own indexing up and running.

Excuse the self-promotion but Mojeek offers a web search API (>9 billion pages): https://www.mojeek.com/services/search/web-search-api/

Nice! I hope Kagi is licensing access from you?

Re: Updates to our web search products and Programmable Search Engine capabilities

#182

Post by saltysalt »

1718627440 wrote: post #135
saltysalt wrote: post #124

Earlier quoted context omitted.

Great feedback, agree I need to filter here. Some website localization is very hard to work around, because they will try to geo-locate the IP address of your bot and redirect it accordingly to a given language...

The issue I was having was with the query "term+wikipedia" it then shows the wikipedia article in Czech, Hungarian, Russian, some kind of Arab and other before finally showing the English version. Then also a lot of that occur 2,3,4+ times with the same URL, just differing in crawltime by a few minutes.

It's a difficult problem to fix, you can set an Accept-Language header on crawl requests but his only works if the target website uses "Content Negotiation." Some sites ignore headers and determine language based on the IP address (Geo-IP) or the URL structure (e.g., /es/ vs /en/), basically a mess...

Re: Updates to our web search products and Programmable Search Engine capabilities

#183

Post by consumer451 »

Terretta wrote: post #152
consumer451 wrote: post #111

Dumb question: I keep seeing posts about how ~"the volume of AI scrapers is making hosting untenable." There must a ton of new full-web datasets out there, right? What are the major hurdles that prevent the owners of these datasets from providing them to third parties via API? Is it the quality of SERP, or staleness? Otherwise, this seems like a potentially lucrative pivot/side hustle?

> the volume of AI scrapers is making hosting untenable Aside from that potential, it's also not true. A Pentium Pro or PIII SSE with circa 1998-99 Apache happily delivers a billion hits a month w/o breaking a sweat unless you think generating pages for every visit is better than generating pages when they change.

Please see my reply to the other child comment. That is my actual question, apologies for not being more clear.

Re: Updates to our web search products and Programmable Search Engine capabilities

#184

Post by 1718627440 »

saltysalt wrote: post #159
1718627440 wrote: post #158

Earlier quoted context omitted.

Sad state the web is in. It is intended, that the page currently shows a link to the wordpress login?

It does not use WordPress.

I'm sorry, I am dumb and visited http://grepper.org/ . Where does your name come from I guess from grep for the WWW?

Re: Updates to our web search products and Programmable Search Engine capabilities

#185

Post by Ferret7446 »

DangitBobby wrote: post #120
jsnell wrote: post #94

Earlier quoted context omitted.

They then go on to say that they pay a 3rd party company to scrape Google results (and serve those scraped results to their users). So their search engine is indeed based on unauthorized and uncompensated use of Google's index. But since they're not using/paying for a supported API but just taking what they want, they indeed are unlikely to be impacted by this API turndown.

Congrats on saying that in the most one-sided way possible. Google makes it literally impossible for them to pay for access to search results to make the product they want (customizable subscription search with no ads), and Google also is the de-facto globally sanctioned crawler because they are the only search engine anyone gives a shit about, and also sites need to be indexed by them to survive. In short, Google ow…

They have a monopoly on their own search results. There's nothing stopping anyone from making their own (hell, a poster did so in the comments above). God forbid we aren't entitled access to the fruits of their labor; the reason you want it isn't because you can't make it (again, see above). It's because making it good is hard, and you want the good results without yourself putting in the effort to make it

Re: Updates to our web search products and Programmable Search Engine capabilities

#186

Post by 1718627440 »

saltysalt wrote: post #182
1718627440 wrote: post #135

Earlier quoted context omitted.

The issue I was having was with the query "term+wikipedia" it then shows the wikipedia article in Czech, Hungarian, Russian, some kind of Arab and other before finally showing the English version. Then also a lot of that occur 2,3,4+ times with the same URL, just differing in crawltime by a few minutes.

It's a difficult problem to fix, you can set an Accept-Language header on crawl requests but his only works if the target website uses "Content Negotiation." Some sites ignore headers and determine language based on the IP address (Geo-IP) or the URL structure (e.g., /es/ vs /en/), basically a mess...

I don't get the problem you claim. You crawl something and get a document in whatever language the site delivers you. You know the language of that document with the lang=... attribute of the document. What results you show for a given language is under your control and not influenced by what the crawled site chose to serve to the crawler.

Re: Updates to our web search products and Programmable Search Engine capabilities

#188

Post by direwolf20 »

Hackbraten wrote: post #150
TiredOfLife wrote: post #140

Earlier quoted context omitted.

Kagi is backed by russia so they will be fine.

You’re probably referring to the fact that their search results include entries from Yandex. That’s something entirely different from being “backed by Russia.” If anything, they pay Russia, not the other way around.

And Yandex has better results than Google, so I support this move. The USA does more wars than Russia does anyway.

Re: Updates to our web search products and Programmable Search Engine capabilities

#189

Post by arantius »

consumer451 wrote: post #179
Tenemo wrote: post #166

Earlier quoted context omitted.

I think it is true that it is a real problem (EDIT: but doesn't necessarily make "hosting untenable"), but you are correct to point out that modern pages tend to be horribly optimized (and that's the source of the problem). Even "dynamic" pages using React/Next.js etc. could be pre-rendered and/or cached and/or distributed via CDNs. A simple cache or a CDN should be enough to handle pretty much any scrapping traffic…

OK, I agree with both of you. I am an old who is aware of NGINX and C10k. However, my question is: what are the economic or technical difficulties that prevent one of these new web-scale crawlers from releasing og-pagerank-api.com? We all love to complain about modern Google SERP, but what actually prevents that original Google experience from happening, in 2026? Is it not possible? Or, is that what orgs like Perplex…

What prevents it is that the web in 2026 is very different than it was when OG pagerank became popular (because it was good). Back then, many pages linked to many other pages. Now a significant amount of content (newer content, which is often what people want) is either only in video form, or in a walled garden with no links, neither in or out of the walls. Or locked up in an app, not out on the general/indexable/linkable web. (Yes, of course, a lot of the original web is still there. But it's now a minority at best.)

Also, of course, the amount of spam-for-SEO (pre-slop slop?) as a proportion of what's out there has also grown over time.

IOW: Google has "gotten worse" because the web has gotten worse. Garbage in, garbage out.

Re: Updates to our web search products and Programmable Search Engine capabilities

#190

Post by consumer451 »

arantius wrote: post #189
consumer451 wrote: post #179

Earlier quoted context omitted.

OK, I agree with both of you. I am an old who is aware of NGINX and C10k. However, my question is: what are the economic or technical difficulties that prevent one of these new web-scale crawlers from releasing og-pagerank-api.com? We all love to complain about modern Google SERP, but what actually prevents that original Google experience from happening, in 2026? Is it not possible? Or, is that what orgs like Perplex…

What prevents it is that the web in 2026 is very different than it was when OG pagerank became popular (because it was good). Back then, many pages linked to many other pages. Now a significant amount of content (newer content, which is often what people want) is either only in video form, or in a walled garden with no links, neither in or out of the walls. Or locked up in an app, not out on the general/indexable/lin…

Thanks for the reply. I mentioned tech, but forgot about time. Yeah, that makes solid sense.

> Or locked up in an app...

I believe you may have at least partially meant Discord, for which I personally have significant hate. Not really for the owners/devs, but why in the heck would any product owner want to hide the knowledge of how to user their app on a closed platform? No search engine can find it, no LLM can learn from it(?). Lost knowledge. I hate it so much. Yes, user engagement, but knowledge vs. engagement is the battle of our era, and knowledge keeps losing.

r/anything is so much better than a Discord server, especially in the age of "Software 3.0"

Post reply on HN