Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude can now search the web

anthropic.com

Post reply on HN
Print view View original

191–200 of 758 posts

Re: Claude can now search the web

#191

Post by webstrand »

sumeno wrote: post #147
Beijinger wrote: post #133

Earlier quoted context omitted.

Is google search bad? Click here to find ten reasons why it is bad and 10 reasons why you should still use it. Yes, it is that bad. Website of Nike? Website of Starbucks? Likely position number one. Every product, category etc., e.g. what rice cooker should I buy? Is diseased by link and affiliate spam. There is a reason why people put +reddit on search terms.

"what rice cooker should I buy" returns a pretty in depth article on bonappetit.com and reddit as the top results, both recommending Zojirushi

Well first Zojirushi is unnecessarily expensive and difficult to clean. Only if you need its fancy options and like multiple varieties of rice would I recommend it. Reddit is no panacea to spam, these days.

But bonappetit.com is exactly an example of affiliate link spam. Even their budget option is awful.

Re: Claude can now search the web

#192

Post by soulofmischief »

bayindirh wrote: post #69
josh-sematic wrote: post #54

Earlier quoted context omitted.

Conversely, every browser is a program that automatically executes HTTP requests.

Yet they respect a lot of things meant for machine to machine interaction. Like server return codes, cookie negotiations, and CAPTCHAs if they behave a certain way. So they sometimes hit bollards and turnstiles made for other types of code which executes HTTP requests. So they're bots basically, but better (or suitably) behaving ones.

Browsers let you visit websites without regard for robots.txt.

Re: Claude can now search the web

#193

Post by dcre »

simonw wrote: post #165
douglee650 wrote: post #155

You know what Claude can't do? Successfully vibe me a solution to: """ i need a bashrc command that will map the alias "logg" to open macvim to the file at ~/log.txt, then execute the macro defined by " z" """ Note z ends with user in insert mode, Claude provides solution below but puts me in edit mode. (I still have to press "i") alias loggg='mvim ~/log.txt -c "normal \ z"'

I don't find that prompt to be particularly clear.

And it did exactly what the prompt asked! For all it knows, the macro could put the user in the desired mode.

Re: Claude can now search the web

#194

Post by MattSayar »

gcanyon wrote: post #177

Funny, I literally just two days ago asked Claude to provide an outline of the functionality of a product, giving it the web site. It of course refused. So I downloaded the text of the site and passed that in, and got mediocre results. The results based on giving the source URL directly were better. Still a bit generic and high-level and vague, as LLMs tend to be, but better than the text-download version a couple da…

I had tried using monolith [0] to feed webpages into Claude but all the html was too much token context. I ended up Print > Save as PDF-ing somewhat often and that worked pretty well. But just giving a URL is ideal.

[0] https://github.com/Y2Z/monolith

Re: Claude can now search the web

#195

Post by verdverm »

SoftTalker wrote: post #76
NBJack wrote: post #10

I wonder if it will actually respect the robots.txt this time.

Maybe we need a new "ai.txt" that says "yes I mean you, ChatGPT et. al."

Bluesky / ATProto has a proposal for User Intents for data. More semantics than robots.txt, but equally unenforceable. Usage with AI is one of the intents to be signaled by users

https://github.com/bluesky-social/proposals/tree/main/0008-u...

Re: Claude can now search the web

#196

Post by pixl97 »

losteric wrote: post #83
1shooner wrote: post #71

Earlier quoted context omitted.

>You can now use Claude to search the internet to provide more up-to-date and relevant responses. It's a search engine. You 'ask it to read the web' just like you asked Google to, except Google used to actually give the website traffic. I appreciate the concept of an AI User-agent, but without a business model that pays for the content creation, this is just going to lead to the death of anonymously accessible conten…

What was the web like before wide spread internet ads, auth, and search engines? Did all those old sites have “business models”? What did the web feel like back then? (This is rhetorical - I had niche hobby sites back then, in the same way some people put out free zines, and wouldn’t give a damn about today’s AI agents so long as they were respectful. The web was better back then, and I believe AI slop and agents bri…

The web was so much smaller back then. Just imagine I turned the user (not automated in any way) based clicks that can occur from a link like reddit today towards your site then. We called it the slashdot effect way back, but that many clicks might take down the entire ISP.

Many of these sites business model was simply "don't cost too much". The moment the web got big a lot of these sites died. Now add DDOS for fun and profit became a thing, most people moved to huge advertising based providers/hosters (think FB).

Simply put, we're never getting the old web back. Now, we may get something new, but it will be different and still far more commercial.

Re: Claude can now search the web

#198

Post by renewiltord »

usrbinbash wrote: post #148
_2d30 wrote: post #39

Earlier quoted context omitted.

I don't think it should. If a user asks the AI to read the web for them, it should read the web for them. This isn't a vacuum charged with crawling the web, it's an adhoc GET request.

> This isn't a vacuum charged with crawling the web, it's an adhoc GET request. Doesn't matter. The robots-exclusion-standard is not just about webcrawlers. A `robots.txt` can list arbitrary UserAgents. Of course, an AI with automated websearch could ignore that, as can webcrawlers. If they chose do that, then at some point, some server admins might, (again, same as with non-compliant webcrawlers), use more drastic m…

In the limit of the arms race it's sufficient for the robot to use the user's local environment to do the browsing. At that point you can't distinguish the human from the robot.

Re: Claude can now search the web

#199

Post by terribleperson »

webstrand wrote: post #191
sumeno wrote: post #147

Earlier quoted context omitted.

"what rice cooker should I buy" returns a pretty in depth article on bonappetit.com and reddit as the top results, both recommending Zojirushi

Well first Zojirushi is unnecessarily expensive and difficult to clean. Only if you need its fancy options and like multiple varieties of rice would I recommend it. Reddit is no panacea to spam, these days. But bonappetit.com is exactly an example of affiliate link spam. Even their budget option is awful.

Zojirushi being difficult to clean is kind of a wild take, unless most of their other rice makers are a wild departure from mine.

Re: Claude can now search the web

#200

Post by pixl97 »

jrflowers wrote: post #103
TheDudeMan wrote: post #78

Earlier quoted context omitted.

But this isn't automated. This is user-driven.

If this feature isn’t already part of the Claude API it likely will be at some point, in which case many Claude requests will be automated with no way to distinguish between user-driven or otherwise.

Simply put, at the end of the day you lose, AI blocking will not work.

I mean, currently the AI request comes from the datacenter running the AI, but eventually one of two things will happen.

AI models will get small/fast enough to run on user hardware and use the users resources: End result? You lose. The user will set their own headers and sites will play the impossible game of identifying AI.

AI sites will figure out how to route the requests via any number of potential methods so the requests appear to come from the user anyway: End result? You lose. The sites attempting to block will play the cat and mouse game of figuring out what is AI or not AI.

Note, this doesn't mean AI blocking isn't worth doing, if nothing else to reduce load on the servers. It's just not a long term winning strategy.

Post reply on HN