So in many respects, search the place it used to construct the model? Isn't that functionally bias-reinforcing? "Look what I synthesise is correct and true because when I use the same top 10 priming responses which informed my decision I find these INDEPENDENT RESULTS which confirm what I modelled" type reasoning. None of us have a problem with an LLM which returns 2+2 = 4 and shows you 10 sites which confirm. What w…
We will very quickly enter a Kepler effect of information on the internet. All text on the internet will become AI slop being parsed by AI. Real information and human beings will be drowned out by the garbage. The internet will cease to be useful and we will retreat to corners of the web or to walled gardens. I'm seeing more and more online communities these days enforce invite only because there's just too much AI s…
Claude can now search the web
511–520 of 758 posts
Re: Claude can now search the web
#512Earlier quoted context omitted.
I don't think it should. If a user asks the AI to read the web for them, it should read the web for them. This isn't a vacuum charged with crawling the web, it's an adhoc GET request.
Many if not most websites are paid for by eyeballs not by get requests. A bot is a bot is a bot. Respect robots.txt or expect to have your IPs banned.
robots.txt is not a security mechanism, and it doesn’t “control bots.” It’s a voluntary convention mainly followed by well behaved search engine crawlers like Google and ignored by everything else.
If you’re relying on robots.txt to prevent access from non human users, you’re fundamentally misunderstanding its purpose. It’s a polite request to crawlers, not an enforcement mechanism against any and all forms of automated access.
Re: Claude can now search the web
#513So, referring specifically to the example they show on the front-page, what value does this bring actually? The best example they could come up with is Typescript migration ? Really? Weren't the LLMs supposed to be a superior alternative to searching the web? Why do we need to produce more CO2 to do the same we could have done at the fraction of the cost, of course at the time when the google search was still working…
The CO2 concerns of using LLMs are massively overblown these days (with the exception of o1-pro and GPT-4.5 at least). The energy efficiency of most models has improved by an order of magnitude since the most widely cited CO2 usage papers were published. (It remains frustratingly difficult to get accurate numbers though: at this point I think more transparency would help rather than hurt the big AI labs)
Re: Claude can now search the web
#514Earlier quoted context omitted.
"and believe my word for it - it was much better back then than the crap you get out of torturing whatever your LLM of choice is" I was around as well and my memories do not confirm this. But google search definitely degraded a lot.
Yes and no. You used to find niche websites more easily, but I vividly remember the frustration with ExpertsExchange results (with answers that were all paywalled).
Re: Claude can now search the web
#515Earlier quoted context omitted.
I don't think it should. If a user asks the AI to read the web for them, it should read the web for them. This isn't a vacuum charged with crawling the web, it's an adhoc GET request.
No thank you, when I define a robots.txt file I expect all automated systems to respect it.
Absolutely nothing has to obey robots.txt. It’s a politeness guideline for crawlers, not a rule, and anyone expecting bots to universally respect it is misunderstanding its purpose.
Re: Claude can now search the web
#516Earlier quoted context omitted.
[flagged]
I don’t respond well to peer pressure. It makes me sick to the stomach. Peer pressure from aggressive behaviour is ironically how Germany’s population got talked into committing genocide. I’ll start doing what other people say for no good reason the day I switch off my brain.
Re: Claude can now search the web
#517Searching the web is a great feature in theory, but every implementation I've used so far looks at the top X hits and then interprets it to be the correct answer. When you're talking to an LLM about popular topics or common errors, the top results are often just blogspam or unresolved forum posts, so the you never get an answer to your problem. More of an indicator that web search is more unusable than ever, but inte…
> web search is more unusable than ever I’m curious why I’m seeing a lot of people thinking this lately. Google definitely made the algorithm worse for customers and better for ads, but I’m almost always able to find what I’m looking for in the working day still. What are other people’s experiences?
Re: Claude can now search the web
#518Earlier quoted context omitted.
Purely on its technical merits Grok is pretty good and fills a niche in the selection of AI agents. But I can absolutely understand not wanting to use an AI owned by somebody who makes Nazi salutes and is dismantling the US government.
[flagged]
Re: Claude can now search the web
#519Searching the web is a great feature in theory, but every implementation I've used so far looks at the top X hits and then interprets it to be the correct answer. When you're talking to an LLM about popular topics or common errors, the top results are often just blogspam or unresolved forum posts, so the you never get an answer to your problem. More of an indicator that web search is more unusable than ever, but inte…