111–120 of 207 posts
I keep seeing posts about how ~"the volume of AI scrapers is making hosting untenable."
There must a ton of new full-web datasets out there, right?
What are the major hurdles that prevent the owners of these datasets from providing them to third parties via API? Is it the quality of SERP, or staleness? Otherwise, this seems like a potentially lucrative pivot/side hustle?
Earlier quoted context omitted.
They can build whatever they want with lots of #hashtags and public money, but that doesn't mean they'll get 30% of French people to use it. But of course they managed to cut themselves a nice salary with EU funds, paid in part by me and you, so that's all that matters.
> with lots of #hashtags I missed this one. What was it about?
Earlier quoted context omitted.
Better late than never. > The French index is at an advanced stage of completion, we have started creating the German language index, and the English one should start shortly. All progress is quickly integrated into the Qwant STAAN API. https://noc.social/@327ppm/115934198650900394
They can build whatever they want with lots of #hashtags and public money, but that doesn't mean they'll get 30% of French people to use it. But of course they managed to cut themselves a nice salary with EU funds, paid in part by me and you, so that's all that matters.
If they deem it necessary to rein in Google, they will rein in Google. There's no lack of tools for this, ranging from obliging phones sold in French territory to offer the French search engine as the default, to forcing every Google search result to promote the local search engine prominently, to campaigns about how it's important for national security not to rely on an adversary/enemy country's services, to everything in between and beyond.
Earlier quoted context omitted.
Anchor text information is arguably a better source for relevance ranking in my experience. I publish exports of the ones Marginalia is aware of[1] if you want to play with integrating them. [1] https://downloads.marginalia.nu/exports/ grab 'atags-25-04-20.parquet'
Though I'd think that you'd want to weight unaffiliated sites' anchor text to a given URL much higher than an affiliated site. "Affiliation" is a tricky term itself. Content farms were popular in the aughts (though they seem to have largely subsided), firms such as Claria and Gator. There are chumboxes (Outbrain, Taboola), and of course affiliate links (e.g., to Amazon or other shopping sites). SEO manipulation is it…
I've also seen some benefit fingerpinting the network traffic the websites make using a headless browser, to identify which ad networks they load. Very few spam sites have no ads, since there wouldn't be any economy in that.
e.g. https://marginalia-search.com/site/www.salon.com?view=traffi...
The full data set of DOM samples + recorded network traffic are in an enormous sqlite file (400GB+), and I haven't yet worked out any way of distributing the data yet. Though it's in the back of my mind as something I'd like to solve.
The 'Google Graveyard is real' sentiment captures something important: every dependency on a large platform is a loan that can be called in. The 34-million-document indie index project someone mentioned is the right response - own your core infrastructure. Easier said than done for whole-web search, but the same principle applies everywhere.
Much easier said than done, especially if you are serving users on scale.
Earlier quoted context omitted.
Good luck scraping websites without being blocked, if you're not Google.
Well you'll get blocked some places but it's not too big of a deal. If you're running an above board operation, you can surprisingly often successfully just email the admin explaining what you're doing, and ask to be unblocked.
Earlier quoted context omitted.
PageRank was an innovative idea in the early days of the Internet when trust was high, but yes it's absolutely gamed now and I would be surprised if Google still relies on it. Fair play to them though, it enabled them to build a massive business.
Google’s biggest search signal now is aggregate behavioral data reported from Chrome. That pervasive behavioral surveillance is the main reason Apple has never allowed a native Chrome app on iOS. It’s also why it is so hard to compete with Google. You guys are talking about techniques for analyzing the corpus of the search index. Google does that and has a direct view into how millions of people interact with it.
The Chrome iOS app still knows every url visited, duration, scroll depth, etc.
Earlier quoted context omitted.
PageRank was an innovative idea in the early days of the Internet when trust was high, but yes it's absolutely gamed now and I would be surprised if Google still relies on it. Fair play to them though, it enabled them to build a massive business.
Google’s biggest search signal now is aggregate behavioral data reported from Chrome. That pervasive behavioral surveillance is the main reason Apple has never allowed a native Chrome app on iOS. It’s also why it is so hard to compete with Google. You guys are talking about techniques for analyzing the corpus of the search index. Google does that and has a direct view into how millions of people interact with it.
Earlier quoted context omitted.
Kagi does not use Google's search index. From their post which made the front page of HN yesterday [1]: > Google does not offer a public search API. The only available path is an ad-syndication bundle with no changes to result presentation - the model Startpage uses. Ad syndication is a non-starter for Kagi’s ad-free subscription model. [1]: https://news.ycombinator.com/item?id=46708678
They then go on to say that they pay a 3rd party company to scrape Google results (and serve those scraped results to their users). So their search engine is indeed based on unauthorized and uncompensated use of Google's index. But since they're not using/paying for a supported API but just taking what they want, they indeed are unlikely to be impacted by this API turndown.