I don't really see any mentioning of a new API, beyond their Vertex AI thing, and I don't know how comparable that might be. Also it is capped at 50 domains (by default). It is perhaps a clever legal workaround. They must sell access to their index, but the verdict didn't state how much of it you can buy access to at any one time. So they put a limit of 50 domains, because that accommodates everyone who's not a searc…
It's only a "clever" workaround in a captured legal system that isn't interested in anti-monopoly outcomes. Any competent legal system would slap that shit down. Just the thought that they could "hack this ruling with one weird trick" is infuriating.
Google quietly announced that Programmable Search (ex-Custom Search) won’t allow new engines to “search the entire web” anymore. New engines are capped at searching up to 50 domains, and existing full-web engines have until Jan 1, 2027 to transition. If you actually need whole-web search, Google now points you to an “interest form” for enterprise solutions (Vertex AI Search etc.), with no public pricing and no guaran…
I know that duckduckgo uses Microsoft Bing Custom search and honestly it is a much more robust system since you don't have to worry about Google axing it. https://www.customsearch.ai
Bing Custom Search was discontinued last year. Although duckduckgo probably has some kind of special contract with Microsoft.
Instead you worry about Microsoft axing it? Sure, it might take 3 years instead of 6 months, and the shutdown period would be 1 year instead of 1 month, but hardly either are long-term solutions.
> it might take 3 years instead of 6 months, and the shutdown period would be 1 year instead of 1 month This matters much more than people (and evidently those within Google) realize
It matters more the shorter your future planning is. Neither works if you're looking forward 3-4 years, for example.
Google quietly announced that Programmable Search (ex-Custom Search) won’t allow new engines to “search the entire web” anymore. New engines are capped at searching up to 50 domains, and existing full-web engines have until Jan 1, 2027 to transition. If you actually need whole-web search, Google now points you to an “interest form” for enterprise solutions (Vertex AI Search etc.), with no public pricing and no guaran…
If its Motion for a Partial Stay is denied, or if it loses on appeal, then under this Final Judgement Google will be forced to offer syndicated "full web" search to Qualified Competitors
Dumb question: I keep seeing posts about how ~"the volume of AI scrapers is making hosting untenable." There must a ton of new full-web datasets out there, right? What are the major hurdles that prevent the owners of these datasets from providing them to third parties via API? Is it the quality of SERP, or staleness? Otherwise, this seems like a potentially lucrative pivot/side hustle?
> the volume of AI scrapers is making hosting untenable Aside from that potential, it's also not true. A Pentium Pro or PIII SSE with circa 1998-99 Apache happily delivers a billion hits a month w/o breaking a sweat unless you think generating pages for every visit is better than generating pages when they change.
I think it is true that it is a real problem (EDIT: but doesn't necessarily make "hosting untenable"), but you are correct to point out that modern pages tend to be horribly optimized (and that's the source of the problem). Even "dynamic" pages using React/Next.js etc. could be pre-rendered and/or cached and/or distributed via CDNs. A simple cache or a CDN should be enough to handle pretty much any scrapping traffic unless you need to do some crazy logic on every page visit – which should almost never be the case on public-facing sites. As an example, my personal site is technically written in React, but it's fully pre-rendered and doesn't even serve JS – it can handle huge amounts of bot/scrapping traffic via its CDN.
Google quietly announced that Programmable Search (ex-Custom Search) won’t allow new engines to “search the entire web” anymore. New engines are capped at searching up to 50 domains, and existing full-web engines have until Jan 1, 2027 to transition. If you actually need whole-web search, Google now points you to an “interest form” for enterprise solutions (Vertex AI Search etc.), with no public pricing and no guaran…
I built my own web search index on bare metal, index now up to 34m docs: https://greppr.org/ People rely too much on other people's infra and services, which can be decommissioned anytime. The Google Graveyard is real.
That's super cool! Do you have any plans to commercialize it or it's just a pet project?
Congrats on saying that in the most one-sided way possible. Google makes it literally impossible for them to pay for access to search results to make the product they want (customizable subscription search with no ads), and Google also is the de-facto globally sanctioned crawler because they are the only search engine anyone gives a shit about, and also sites need to be indexed by them to survive. In short, Google ow…
>In short, Google owns the river and sells the boats, and the public built a wall around it. That would be a monopoly if there was only 1 river in the whole world.
Yeah I mean think whatever you need to for the metaphor to work.
I built my own web search index on bare metal, index now up to 34m docs: https://greppr.org/ People rely too much on other people's infra and services, which can be decommissioned anytime. The Google Graveyard is real.
That's super cool! Do you have any plans to commercialize it or it's just a pet project?
I'm curious about what it would take to build my own "toy" search engine with its own index. Anyone ever tried this?
Yeah that's where I started out in 2021. Been at it for almost 5 years now, last three of which full time. I'm indexing about 1.1 billion documents now off a single server. Hard part is doing it at any sort of scale and producing useful results. It's easy to build something that indexes a few million documents. Pushing into billions is a bigger challenge, as you start needing a lot of increasingly intricate bespoke s…
I assume you able to monetize it since you work on it full time?