231–240 of 342 posts
Few people realise that virtually everything we do online has, until this point, been free training to make OpenAI, Anthropic, etc. richer while cutting humans--the ones who produced the value--out of the loop. It might be too little, too late, at this juncture, and this particular solution doesn't seem too innovative. However, it is directionally 100% correct, and let's hope for massively more innovation in defendin…
Earlier quoted context omitted.
> OSS > people are making trillions off the free labor of people like you and me I read "No Discrimination Against Fields of Endeavor" to also include LLMs and especially the cases that we most deeply disagree with. Either we believe in the principles of OSS or we do not. If you do not like the idea of your intellectual property being used for commercial purposes then this model is definitely not for you. There is no…
Open source software typically has a license. People not following the license isn’t tolerated. This is what AI scrapers are doing. They’re taking your code, your artwork and your writing without any consideration for the license.
There is an active case on this, where Microsoft has been sued over GitHub copilot, and it has been slowly moving through the court system since 2022. Most of the claims have been dismissed, and the prediction market is at 11%: https://manifold.markets/JeffKaufman/will-the-github-copilot...
Earlier quoted context omitted.
Serious question: You put Cloudflare between all your domains and all your visitors without looking in to how this would affect your site's reachability? If so, that's interesting, considering that many people in this community are negatively affected by Cloudflare because they're using Linux and/or some less than mainstream browser. You might want to read some threads on here about Cloudflare.
Where did I say all. Most of the time I don't use them for their network, usually just DNS records for mail because their interface is nicer than namecheap and gives me basic stats. To my understanding, they aren't blocking MX records behind captchas
I've heard lots of people on HN complaining about bot traffic bogging down their websites, and as a website operator myself I'm honestly puzzled. If you're already using Cloudflare, some basic cache configuration should guarantee that most bot traffic hits the cache and doesn't bog down your servers. And even if you don't want to do that, bandwidth and CPU are so cheap these days that it shouldn't make a difference.…
That's a pretty big assumption. The largest site I work on has 100,000s of pages, each in around 10 languages — that's already millions of pages. It generally works fine. Yesterday it served just under 1000 RPS over the day. AI crawlers have brought it down when a single crawler has added 100, 200 or more RPS distributed over a wide range of IPs — it's not so much the number of extra requests, though it's very dispro…
Earlier quoted context omitted.
Open source software typically has a license. People not following the license isn’t tolerated. This is what AI scrapers are doing. They’re taking your code, your artwork and your writing without any consideration for the license.
Weather training on code is fair use is still an open legal question, and it may well be fair use. The way a license works is by saying "you have my permission to use this code as long as you follow these conditions", but if no license is required than the conditions are irrelevant. There is an active case on this, where Microsoft has been sued over GitHub copilot, and it has been slowly moving through the court syst…
I'm putting my new code somewhere private anyway.
I'm still not sure this is going to be very effective, as so many of the worst offenders don't identify themselves as bots, and often change their user agent. Has Cloudflare said anything about identifying the bad actors?
Yes, they have over the years, for example https://blog.cloudflare.com/residential-proxy-bot-detection-... , https://blog.cloudflare.com/cloudflare-bot-management-machin... , https://blog.cloudflare.com/introducing-bot-analytics/ .
Earlier quoted context omitted.
> I think they have to for ipv6 addresses… there’s just way too many bots and way too many addresses Are you really arguing that it's legitimate to consider all IPv6 browsing traffic "suspicious"? If anything, I'd say that IPv4 is probably harder, given that NATs can hide hundreds or thousands of users behind a single IPv4 address, some of which might be malicious. > you may have gotten an address that was previously…
For a whitelist system, then by definition yes? If it’s a blacklist system, like I said I’ve not heard of any feasible solution more precise than banning huge ranges of ipv6 addresses.
A whitelist system would consider all IPv4 traffic suspicious by default too. This is not an answer to why you'd be suspicious of IPv6 in particular.
> I’ve not heard of any feasible solution more precise than banning huge ranges of ipv6 addresses.
Handling /56s or something like that is about the same as handling individual IPv4 addresses.
Earlier quoted context omitted.
Weather training on code is fair use is still an open legal question, and it may well be fair use. The way a license works is by saying "you have my permission to use this code as long as you follow these conditions", but if no license is required than the conditions are irrelevant. There is an active case on this, where Microsoft has been sued over GitHub copilot, and it has been slowly moving through the court syst…
I can't see how it can be fair use. Just follow the license, it's not that difficult. Microsoft will forever be a pariah if they get away with this. I'm putting my new code somewhere private anyway.
The key question is whether it is sufficiently "transformative". See Authors Guild vs Google, Kelly vs Arriba Soft, and Sony vs Universal. This is a way a judge could definitely rule, and at this point I think is the most likely outcome.
> Microsoft will forever be a pariah if they get away with this.
I doubt this. Talking to developers, it seems like the majority are pretty excited about coding assistants. Including the ones that many companies other than Microsoft (especially Anthropic) are putting out.
Sounds very basic, sadly. Anybody know why these web crawling/bot standards are not evolving ? I believe robots.txt was invented in 1994(thx chatgpt). People have tried with sitemaps, RSS and IndexNow, but its like huge$$ organizations are depending on HelloWorld.bas tech to control their entire platform. I want to spin up endpoints/mcp/etc. and let intelligent bots communicate with my services. Let them ask for acce…
robots.txt was invented in 1994(thx chatgpt) Thought of and discussed as a possibility in 1994. Proposed as a standard in 2019. Adopted as a standard in 2022. Thanks, IETF.
Earlier quoted context omitted.
I can't see how it can be fair use. Just follow the license, it's not that difficult. Microsoft will forever be a pariah if they get away with this. I'm putting my new code somewhere private anyway.
> I can't see how it can be fair use. The key question is whether it is sufficiently "transformative". See Authors Guild vs Google, Kelly vs Arriba Soft, and Sony vs Universal. This is a way a judge could definitely rule, and at this point I think is the most likely outcome. > Microsoft will forever be a pariah if they get away with this. I doubt this. Talking to developers, it seems like the majority are pretty exci…