41–50 of 279 posts
I don’t know why anyone would still be trying to pull data off the open internet. Too much signal to noise. So much AI influence already baked into the corpus. You are just going to be reinforcing existing bias. I’m more worried about the day Amazon or Hugging Face take down their large data sets.
Cloudflare has a service for this now that will detect AI scrapers and send them to a tarpit of infinite AI generated nonsense pages.
Wow, so to prevent AI scrapers from harvesting my data I need to send all of my traffic through a third party company that gets to decide who gets to view my content. Great idea!
Its not just AI scrappers doing it by themselves but now users are also being trained to put the link in the claude chat/chat gpt and ask it to summarise it. And off course that would show up on the website end as a scraper. In fact firefox now allows you to preview the link and get key points without ever going to the link[1] [1] https://imgur.com/a/3E17Dts
> [1] https://imgur.com/a/3E17Dts
This is generated on device with llama.cpp compiled to webassembly (aka wllama) and running SmolLM2-360M. [1] How is this different from the user clicking on the link? In the end, your local firefox will fetch the link in order to summarize it, the same way you would have followed the link and read through the document in reader mode.
[1] https://blog.mozilla.org/en/mozilla/ai/ai-tech/ai-link-previ...
We should add optional `tips` addresses in llms.txt files.
We're also working on enabling and solving this at Grove.city.
Human Agent Human Tips don't account for all the edge cases, but they're a necessary and happy neutral medium.
Moving fast. Would love to share more with the community.
Wrote about it here: https://x.com/olshansky/status/2008282844624216293
I wonder if a model similar to this (but decentralized/federated or something) could be used to help fight bots?
I feel the pain — it’s very difficult to detect many of the less ethical scrapers. They use residential IP pools, rotate IPs, and provide valid user agents.
And many proxy/scraping providers now use real browsers that can automatically bypass cloudflare captchas as well, and a bot with a real browser similarly won't be clicking on any invisible links, so... I am skeptical just how long this will make an appreciable difference.
Resurfacing a proposal I put out on llms-txt: https://github.com/AnswerDotAI/llms-txt/issues/88 We should add optional `tips` addresses in llms.txt files. We're also working on enabling and solving this at Grove.city. Human Agent Human Tips don't account for all the edge cases, but they're a necessary and happy neutral medium. Moving fast. Would love to share more with the community. Wrote about it here: https://x.co…
I can't see this working.
a) Have a reverse proxy that keeps a "request budget" per IP and per net block, but instead of blocking requests, causing the client to rotate their IP, the requests get throttled/slowed down, without dropping them.
b) Write your API servers in more efficient languages. According to their Github, their backend runs on Perl and Python. These technologies have been "good enough" for quite some time, but considering current circumstances and until a better solution is found, this may not be the case anymore and performance and cpu cost per request does matter these days.
c) Optimize your database queries, remove as much code as possible from your unauthenticated GET request handlers, require authentication for the expensive ones.