251–260 of 317 posts
There’s a tiny but important difference between scraping and getting work done. Scraping is associate with mindless extraction. Like a vacuum cleaner sucking-in any data without context, permission, or contributing value back. On the other hand AI agents aren’t here to scrape for the sake of it - I have seen it first hand. They are here to get work done, mostly researching, summarizing, assisting, building new produc…
Hey Orlie.
IMO this is why this will not work. If you're too small a publisher, you don't want to lose potential click-through traffic. If you're a big publisher, you negotiate with the main bots that crawl a site (Perplexity, ChatGPT, Anthropic, Google, Grok).
The only way I can see something like this work is if a large "bot" providers set the standard and say they'll pay if this is set up (unlikely) or smaller apps that crawl see that this as cheaper than a proxy. But in the end, most of the traffic comes from a few large players.
This is cool but I don't like how this forces all crawlers to use Cloudflare. Google Chrome developers were proposing some Web Monetization API in Chromium a few years back, back when the Manifest V3 drama was still fresh, so maybe we should look into that instead. To allow decentralized payments to not be dependent on a single vendor.
Earlier quoted context omitted.
It's not an ideal solution because any fixed cost solution is begging for a middle man/reseller to be introduced. Like I pay the $5 monthly flat fee (or $500, $5k, $500k, whatever it's known fixed cost for me) for the system, turn around and resell all content for a $1 monthly flat fee. There is a real cost to the content you're consuming with that flat-fee. So either the flat fee is more of "credit" system or it's r…
I suppose "credit system" is indeed more accurate than "fee", it's just that I personally would set it at a flat rate and then stop thinking about it, so it would feel like a sort of admission-to-the-internet to me. As for bandwidth and storage costs... that could just be rolled into the same attribution/payment scheme. If content is not propagating well because too few people are hosting it, then I'm ok with allocat…
> It's just that I personally would set it at a flat rate and then stop thinking about it, so it would feel like a sort of admission-to-the-internet to me.
That doesn't matter. A credit system is like an hourly changing flat fee. it doesn't make sense. You might set it at $10 a month, that's it for you. But where is that number coming from. What if you watch a "Just released" movie that costs $10 credits on the first day of the month. No internet for you for the rest of the month? You used to read 10 articles every month, but now $10 you can only read 2. Is that ok? it's a flat fee after all.
> If I notice that I've got a leacher problem (whether as a creator or as a distributor) then maybe I revoke trust in the leachers and they stop getting the content from me.
In other words: "If I notice a bad actor, I block them" congratulations, you have solved all of the internet problems. That idea could be worth billions. Personally I just don't write bugs to begin with and therefore bad actors can't exploit them.
Earlier quoted context omitted.
> An internet with less ads, more genuine information. If AI will lead to this next phase of the internet How is AI supposed to create an internet "with more genuine information", based on what we have seen so far? These two statements appear to be mutually exclusive.
If I understand correctly, it will be not by creating a new iteration, but by destroying the current one.
Earlier quoted context omitted.
Sadly they seem to be getting through this one lately. Had a scraper hitting me with 80 qps punching straight through Anubis. Had to set up a global rate limit that browned out the functionality they were interested in[1] in excessive load. [1] This form https://marginalia-search.com/site/news.ycombinator.com
Probably headless chrome. I'm going to investigate.
Is there anything I can extract, fingerprint wise, if they come back to work out what's going on?
"AI" is certainly creating problems that can be monetized. So humans get the Cloudflare captchas in order to access their own content at Stackoverflow and Mathoverflow, and "AI" crawlers get the data highway for a fee. And all of this does not stop the incumbents who have already stolen everything.
Cloudflare has not used CAPTCHAs since 2023: https://blog.cloudflare.com/turnstile-ga/
https://www.techradar.com/pro/security/fake-cloudflare-captc...
Earlier quoted context omitted.
Because Visa doesn't (hasn't) wanted to do microtransactions, ever, since the beginning of the internet.
You still don't need cryptocurrencies. You just need a middleman that aggregates micropayments into large enough amounts to work with non-micropayment systems. Some might object to having to get middlemen involved, but the thing is that even with cryptocurrency payments you are going to need middlemen because the web is international. If your website is directly charging crawlers to crawl and you get crawled and paid…