Never build a product with core feature depending on a third-party, you will eventually get fucked up for sure. always have a 70:30 rule for revenue where 70% is core independent features.
That's why I eschew HTTPS.
Updates to our web search products and Programmable Search Engine capabilities
71–80 of 207 posts
Re: Updates to our web search products and Programmable Search Engine capabilities
#72Earlier quoted context omitted.
Kagi
> Kagi This seems to be true, but more indirectly. From Kagi’s blog [0] which is a follow up to a Kagi blog post from last year [1]. [0]> Google: Google does not offer a public search API. The only available path is an ad-syndication bundle with no changes to result presentation - the model Startpage uses. Ad syndication is a non-starter for Kagi’s ad-free subscription model.[^1] [0]> The current interim approach (cu…
I believe they try to indirectly say they are using SerpApi or a similar product that scrapes Google search results to use them. And other big ones use it too so it must be ok...
That must be the reason why they limit the searches you can do in the starter plan. Every SerpApi call costs money.
Re: Updates to our web search products and Programmable Search Engine capabilities
#73Earlier quoted context omitted.
I don't weight home pages in any way yet to bump them up, it's just raw search on keyword relevance.
Sure, but the point is results are not relevant at all? It’s cool though, and really fast
What do you mean they're not relevant? The top result you linked contained the word stackoverflow didn't it? It's showing you exactly what you searched for. Why would you need a search engine at all if you already know the name of the thing? Just type stackoverflow.com into your address bar.
I feel like Google-style "search" has made people really dumb and unable to help themselves.
Re: Updates to our web search products and Programmable Search Engine capabilities
#74Is this perhaps to prevent ChatGPT, Claude and Grok to use Google Search? It would make sense for Google to keep that ability for Gemini.
They'll go adversarial interop through SerpAPI, just like Kagi does. SerpAPI will get the money instead of Google getting it.
Re: Updates to our web search products and Programmable Search Engine capabilities
#75Google quietly announced that Programmable Search (ex-Custom Search) won’t allow new engines to “search the entire web” anymore. New engines are capped at searching up to 50 domains, and existing full-web engines have until Jan 1, 2027 to transition. If you actually need whole-web search, Google now points you to an “interest form” for enterprise solutions (Vertex AI Search etc.), with no public pricing and no guaran…
I built my own web search index on bare metal, index now up to 34m docs: https://greppr.org/ People rely too much on other people's infra and services, which can be decommissioned anytime. The Google Graveyard is real.
Lol, a GooglePlus URL was mentionned on a webpage i browsed this week.#blastFromThePast
Re: Updates to our web search products and Programmable Search Engine capabilities
#76Earlier quoted context omitted.
Number of docs isn’t the limiting factor. I just searched for “stackoverflow” and the first result was this: https://www.perl.com/tags/stackoverflow/ The actual Stackoverflow site was ranked way down, below some weird twitter accounts.
I don't weight home pages in any way yet to bump them up, it's just raw search on keyword relevance.
Google's entire (initial) claim-to-fame was "PageRank", referring both to the ranking of pages and co-founder Larry Page, which strongly prioritised a relevance attribute over raw keyword findings (which then-popular alternatives such as Alta Vista, Yahoo, AskJeeves, Lycos, Infoseek, HotBot, etc., relied on, or the rather more notorious paid-rankings schemes in which SERP order was effectively sold). When it was first introduced, Google Web Search was absolutely worlds ahead of any competition. I remember this well having used them previously and adopted Google quite early (1998/99).
Even with PageRank result prioritisation is highly subject to gaming. Raw keyword search is far more so (keyword stuffing and other shenanigans), moreso as any given search engine begins to become popular and catch the attention of publishers.
Google now applies other additional ordering factors as well. And of course has come to dominate SERP results with paid, advertised, listings, which are all but impossible to discern from "organic" search results.
(I've not used Google Web Search as my primary tool for well over a decade, and probably only run a few searches per month. DDG is my primary, though I'll look at a few others including Kagi and Marginalia, though those rarely.)
https://en.wikipedia.org/wiki/PageRank>
"The anatomy of a large-scale hypertextual Web search engine" (1998) http://infolab.stanford.edu/pub/papers/google.pdf> (PDF)
Early (1990s) search engines: https://en.wikipedia.org/wiki/Search_engine#1990s:_Birth_of_...>.
Re: Updates to our web search products and Programmable Search Engine capabilities
#77Earlier quoted context omitted.
I'll work on that adjustment, it's fair feedback thanks!
Unfortunately this is the bulk of search engine work. Recursive scraping is easy in comparison, even with CAPTCHA bypassing. You either limit the index to only highly relevant sites (as Marginalia does) or you must work very hard to separate the spam from the ham. And spam in one search may be ham in another.
I limit it to highly relevant curated seed sites, and don't allow public submissions. I'd rather have a small high-quality index.
You are absolutely right, it is the hardest part!
Re: Updates to our web search products and Programmable Search Engine capabilities
#78This will significantly impact (quite possibly kill) Startpage and Ecosia, who are effectively white-label Google, right? What alternatives are there besides Bing? Is it really so hard that it’s not considered worth doing? Some of the AI companies (Perplexity, Anthropic) seem to have managed to get their own indexing up and running.
Excuse the self-promotion but Mojeek offers a web search API (>9 billion pages): https://www.mojeek.com/services/search/web-search-api/
Re: Updates to our web search products and Programmable Search Engine capabilities
#79Antitrust do not work against large companies. Just dissolve them in acid.
This is the type of monopoly abuse these laws were designed to target, and antitrust laws actually do work against large companies.
If you actually enforce them.
Unfortunately, during the Reagan administration, political sentiment toward monopolies shifted and since then antitrust law has been a paper tiger at best.
Re: Updates to our web search products and Programmable Search Engine capabilities
#80Earlier quoted context omitted.
Sure, but the point is results are not relevant at all? It’s cool though, and really fast
What do you mean they're not relevant? The top result you linked contained the word stackoverflow didn't it? It's showing you exactly what you searched for. Why would you need a search engine at all if you already know the name of the thing? Just type stackoverflow.com into your address bar. I feel like Google-style "search" has made people really dumb and unable to help themselves.
the query is just to highlight that relevance is a complex topic. few people would consider "perl blog posts from 2016 that have the stack overflow tag" as the most relevant result for that query.