The 'Google Graveyard is real' sentiment captures something important: every dependency on a large platform is a loan that can be called in. The 34-million-document indie index project someone mentioned is the right response - own your core infrastructure. Easier said than done for whole-web search, but the same principle applies everywhere.
Updates to our web search products and Programmable Search Engine capabilities
21–30 of 207 posts
Re: Updates to our web search products and Programmable Search Engine capabilities
#22Google quietly announced that Programmable Search (ex-Custom Search) won’t allow new engines to “search the entire web” anymore. New engines are capped at searching up to 50 domains, and existing full-web engines have until Jan 1, 2027 to transition. If you actually need whole-web search, Google now points you to an “interest form” for enterprise solutions (Vertex AI Search etc.), with no public pricing and no guaran…
I built my own web search index on bare metal, index now up to 34m docs: https://greppr.org/ People rely too much on other people's infra and services, which can be decommissioned anytime. The Google Graveyard is real.
Number of docs isn’t the limiting factor.
I just searched for “stackoverflow” and the first result was this: https://www.perl.com/tags/stackoverflow/
The actual Stackoverflow site was ranked way down, below some weird twitter accounts.
Re: Updates to our web search products and Programmable Search Engine capabilities
#23Google quietly announced that Programmable Search (ex-Custom Search) won’t allow new engines to “search the entire web” anymore. New engines are capped at searching up to 50 domains, and existing full-web engines have until Jan 1, 2027 to transition. If you actually need whole-web search, Google now points you to an “interest form” for enterprise solutions (Vertex AI Search etc.), with no public pricing and no guaran…
I built my own web search index on bare metal, index now up to 34m docs: https://greppr.org/ People rely too much on other people's infra and services, which can be decommissioned anytime. The Google Graveyard is real.
I made also something for my own search needs. It's just an SQLite table of domains, and places. I have your search engine there also ;-)
https://github.com/rumca-js/Internet-Places-Database
Demo for most important ones https://rumca-js.github.io/search
Re: Updates to our web search products and Programmable Search Engine capabilities
#24Earlier quoted context omitted.
I built my own web search index on bare metal, index now up to 34m docs: https://greppr.org/ People rely too much on other people's infra and services, which can be decommissioned anytime. The Google Graveyard is real.
I made also something for my own search needs. It's just an SQLite table of domains, and places. I have your search engine there also ;-) https://github.com/rumca-js/Internet-Places-Database Demo for most important ones https://rumca-js.github.io/search
Thank you, will check it out!
Re: Updates to our web search products and Programmable Search Engine capabilities
#25Google quietly announced that Programmable Search (ex-Custom Search) won’t allow new engines to “search the entire web” anymore. New engines are capped at searching up to 50 domains, and existing full-web engines have until Jan 1, 2027 to transition. If you actually need whole-web search, Google now points you to an “interest form” for enterprise solutions (Vertex AI Search etc.), with no public pricing and no guaran…
[deleted]
Re: Updates to our web search products and Programmable Search Engine capabilities
#26Meanwhile in Europe: Qwant and Ecosia team up to build their own search index: https://blog.ecosia.org/eusp/
It's a noble effort, but they're so late to the game that it's hard to see them making a significant dent. I hope I'm wrong. They were: > aiming to serve 30% of French search queries [by end of 2025] https://blog.ecosia.org/launching-our-european-search-index/
I feel like soon there won’t even be a point having a search engine since almost the entire internet will be useless AI slop.
Re: Updates to our web search products and Programmable Search Engine capabilities
#27What examples are there of people using this?
There is literally thousands of independent search engines that use Programmable search to search the entire web. Many ISP providers use it on their homepage, kids-based search engines like wackysafe.com use it, also search engines that focus on privacy like gprivate.com etc
Also LLM tools. Programmable Search Engine API was a way to give third-party LLM frontends the ability to give LLMs a web search tool. Notably, this was a common practice long before any of the major LLM providers added search capabilities to their frontents.
Re: Updates to our web search products and Programmable Search Engine capabilities
#28Earlier quoted context omitted.
What are some of the niche search engines build on Google's index affected by this?
Kagi
No wonder Kagi is angry.
Google is a monopoly across several broad categories. They're also a taxation enterprise.
Google Search took over as the URL bar for 91% of all web users across all devices.
Since this intercepts trademarks and brand names, Google gets to tax all businesses unfairly.
Tell your legislators in the US and the EU that Google shouldn't be able to sell ads against registered trademarks (+/- some edit distance). They re-engineered the web to be a taxation system for all businesses across all categories.
Searching for Claude -> Ads in first place
Searching for ChatGPT -> Ads in first place
Searching for iPhone -> Ads in first place
This is inexcusable.
Only searches for "ChatGPT versus", "iPhone reviews", or "Nintendo game comparison" should allow ads. And one could argue that the "URL Bar" shouldn't auto suggest these either when a trademark is in the URL bar.
If Google won't play fair, we have to kill 50% of their search revenue for being egregiously evil.
If you own a trademark, Google shouldn't be able to sell ads against you.
--
Google's really bad. Ideally we'd get an antitrust breakup. They're worse than Ma Bell. I wouldn't even split Google into multiple companies by division - I'd force them to be multiple copies of the same exact entity that then have to compete with each other:
Bell Systems -> {BellSouth, Bell Atlantic, Southwestern Bell, ...}
Google -> {GoogleA, GoogleB, GoogleC, ...}
They'd each have cloud, search, browser, and YouTube. But new brand names for new parent companies. That would create all-out war and lead to incredible consumer wins.
Re: Updates to our web search products and Programmable Search Engine capabilities
#29The 'Google Graveyard is real' sentiment captures something important: every dependency on a large platform is a loan that can be called in. The 34-million-document indie index project someone mentioned is the right response - own your core infrastructure. Easier said than done for whole-web search, but the same principle applies everywhere.
Much easier said than done, especially if you are serving users on scale.
Re: Updates to our web search products and Programmable Search Engine capabilities
#30I'm curious about what it would take to build my own "toy" search engine with its own index. Anyone ever tried this?
Might find YaCy interesting. It’s meant to be a decentralised search engine where users scrape the internet and can search other users indexes in a kind of torrent like way.
I found it didn’t really work as a real search engine but it was interesting.