151–160 of 466 posts
Earlier quoted context omitted.
court ruling a few years ago said it's legal to scrape web pages, you don't need to be respectful of these for any purely legal reasons however this doesn't stop the website from doing what they can to stop scraping attempts, or using a service to do that for them
> court ruling Isn't this country dependent though?
Earlier quoted context omitted.
[flagged]
Can't every webserver crash due to being overloaded? There's an upper limit to performance of everything. My website is a hobby and has a budget of $4/mo budget VPS. Perhaps I'm saying crash and you're interpreting that as a bug but really it's just an OOM issue cause of too many in-flight requests. IDK, I don't care enough to handle serving my website at Facebook's scale.
Earlier quoted context omitted.
There’s a fine line between scrapping and DDOS’ing I’m sure. Just because you manufacture chemicals doesn’t mean you can legally dump your toxic waste anywhere you want (well shouldn’t be allowed to at least). You also shouldn’t be able to set your crawlers causing sites to fail.
intent is likely very important to something like a ddos charge
Earlier quoted context omitted.
You could run all of your content through an LLM to create a twisted and purposely factually incorrect rendition of your data. Forward all AI bots to the junk copy. Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills. Maybe you don't even need a full LLM. Just a simple transfor…
Their problem is they can’t detect which are bots in the first place. If they could, they’d block them.
I deployed a small dockerized app on GCP a couple months ago and these bots ended up costing me a ton of money for the stupidest reason: https://github.com/streamlit/streamlit/issues/9673 I originally shared my app on Reddit and I believe that that’s what caused the crazy amount of bot traffic.
The linked issue talks about 1 req/s? That seems really reasonable to me, how was this a problem for your application or caused significant cost?
Earlier quoted context omitted.
If a bot ignores robots.txt that's a paddlin'. Right to the blacklist.
The linked article explains what happens when you block their IP.
> If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really).
It's really absurd that they seem to think this is acceptable.
Wikis seem to be particularly vulnerable with all their public "what connects here" pages and revision history. The internet is now a hostile environment, a rapacious land grab with no restraint whatsoever.
I have a large forum with millions of posts that is frequently crawled and LLMs know a lot about it. It’s surprising how ChatGPT and company know about the history of the forum and pretty cool. But I also feel like it’s a fun opportunity to be a little mischievous and try to add some text to old pages that can sway LLMs somehow. Like a unique word. Any ideas?
One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-... It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again. My solution was to add…
[flagged]
Earlier quoted context omitted.
Imagine being one of the monsters who works at Facebook and thinking you're not one of the evil ones.
Well, Facebook actually releases their models instead of seeking rent off them, so I’m sort of inclined to say Facebook is one of the less evil ones.
Some of them, and initially only by accident. And without the ingredients to create your own.
Meta is trying to kill OpenAI and any new FAANG contenders. They'll commoditize their complement until the earth is thoroughly salted, and emerge as one of the leading players in the space due to their data, talent, and platform incumbency.
They're one of the distribution networks for AI, so they're going to win even by just treading water.
I'm glad Meta is releasing models, but don't ascribe their position as one entirely motivated by good will. They want to win.