51–60 of 758 posts
Earlier quoted context omitted.
It must form the search index somehow. That is prior the human action. Simply it would not find the page at all if it respects.
I remember in late 90s/early 2000 as a teen going to robots.txt to specifically see what they were trying to hide and exploring those urls. What is the difference if I use a browser or a LLM tool (or curl, or wget, etc) to make those requests?
Aside, does anyone know of an app like Perplexity for surfing the news in a foreign language (language practice)? Perplexity's "Explore" tab translates its news to your local language, and its curated news items are all pretty interesting, but the problem is that there are so few of them. I seem to get maybe a dozen stories in a day. I paid their subscription for a month just to listen to the news on my walk, but did…
ground.news includes sources from all sorts of countries, and also auto-translate headline and the intro, while you can still click to access the source article. Not affiliated, just happy user.
Example with sources in English, German and French: https://ground.news/article/accident-on-the-a13-in-the-yveli...
Although I'm not sure how useful it is for language learning, as you cannot (afaik) configure it to only display articles in Spanish or something similar, but if you filter by stories about France, you'll get a lot of French sources (obviously).
What's up with the geoblocking of Claude features? Not the first time it happens
As an example, I recently travelled abroad to a popular vacationing spot and asked ChatGPT for local recommendations on what to do. When it gave me answers directly, they were pretty solid. But when it “searched the web” instead, the answers were awful. Every single result it suggested had terrible ratings. It did this repeatedly. One of those times I asked it to pick something with better ratings and it sort of improved but not by much.
Of course this is another tool and maybe Claude uses better sources or a better algorithm, but in this case where there was a concrete number tied to the results, that while not perfect, aims to rate the quality of a result, it still did not filter out low quality answers. I’m not sure I trust these LLMs to do any better when there aren’t such ratings available. The available input data is just not very good, and now LLMs are being used to feed that low quality, SEO machine.
I wonder if it will actually respect the robots.txt this time.
> in feature preview for all paid Claude users in the United States. Support for users on our free plan and more countries is coming soon US only
Earlier quoted context omitted.
almost no one does, robots.txt is practically a joke at this point — right up there with autocomplete=off
In what circles is it a joke? Google bots seem to respect it on my sites according to logs.