451–460 of 466 posts
Earlier quoted context omitted.
The linked issue talks about 1 req/s? That seems really reasonable to me, how was this a problem for your application or caused significant cost?
1 req/s being too much sounds crazy to me. A single VPS should be able to handle hundreds if not thousands of requests per second. For more compute intensive stuff I run them on a spare laptop and reverse proxy through tailscale to expose it
Why do people say things like this? People don't need permission to be helpful in the context of a conversation. If you don't want a conversation, turn off your chat or don't read the chat. If you don't like what they said, move on, or thank them and let them know you don't want it, or be helpful and let them know why their suggestion doesn't work/make sense/etc...
What I don't get is why they need to crawl so aggressively, I have a site with content that doesn't change often (company website) with a few hundred pages total. But the same AI bot will scan the entire site multiple times per day, like somehow all the content is going to suddenly change now after it hasn't for months. That cannot be an efficient use of their money, maybe they used their own AI to write the scraper…
The post mentions that the bots were crawling all the wiki diffs. I think that might be useful to see how text evolves and changes over time. Possibly how it improves over time, and what those improvements are. I guess they are hoping that there will be small changes to your website that it can learn from.
Earlier quoted context omitted.
CommonCrawl archives robots.txt For convenience, you can view the extracted data here: https://pastebin.com/VSHMTThJ You are welcome to verify for yourself by searching for “wiki.diasporafoundation.org/robots.txt” in the CommonCrawl index here: https://index.commoncrawl.org/ The index contains a file name that you can append to the CommonCrawl url to download the archive and view. More detailed information on downloa…
Your attitude is inappropriate and violates the sitewide guidelines for discussion.
Earlier quoted context omitted.
That may all be true. That still doesn’t mean they intentionally lied.
What is the criteria of an intentional lie, then? Admission? The author responded: >denschub 2 days ago [–] >the robots.txt on the wiki is no longer what it was when the bot accessed it. primarily because I clean up my stuff afterwards, and the history is now completely inaccessible to non-authenticated users, so there's no need to maintain my custom robots.txt Which is verifiably untrue: HTTP/1.1 200 server: nginx/1…
There are no “intentional” lies, because there are no “unintentional” lies.
All lies are intentional. An “unintentional lie” is better known as “being wrong”.
Being wrong isn’t always lying. What’s so hard about this? An example:
My wife once asked me if I had taken the trash out to the curb, and I said I had. This was demonstrably false, anyone could see I had not. Yet for whatever reason, I mistakenly believed that I had done it. I did not lie to her. I really believed I had done it. I was wrong.
Earlier quoted context omitted.
Your attitude is inappropriate and violates the sitewide guidelines for discussion.
There are currently two references to “Mangion-ing” OpenAI board members in this thread, several more from Reddit, based on the falsehoods being perpetrated by the author. Is this really someone you want to conspire with? Is calling this out more egregious than the witch hunt being organized here?
If you are legitimately trying to correct misinformation, your attitude, tone and language are counter productive. You would be much better seved by taking that energy and crafting an actually persuasive argument. You come across as unreasonable and unwilling to listen, not someone with a good grasp of the technical specifics.
I don't have a horse in the race. I'm fairly technical, but I did not find your arguments persuasive. This doesn't mean they are wrong, but it does mean that you didn't do a good job of explaining them.
Earlier quoted context omitted.
You will be burning through thousands of dollars worth of compute to do that.
The biggest issue is at least 80% of internet users won’t be capable of passing the test.
Part of the issue, the humans all behaved the same way previously. Just slower.
All the scraping, and web downloading. Humans have been doing that for a long time. Just slower.
It's the same issue with a lot of society. Mean, hurtful humans, made mean hurtful bots.
Always the same excuses too. Company / researchers make horrible excrement, knowing full well its going harm everybody on the world wide web. Then claim they had no idea. "Thoughts and prayers."
The torture that used to exist on the world wide web of copy-pasta pages and constant content theft, is now just faster copy-pasta pages and content theft.
Earlier quoted context omitted.
What is the criteria of an intentional lie, then? Admission? The author responded: >denschub 2 days ago [–] >the robots.txt on the wiki is no longer what it was when the bot accessed it. primarily because I clean up my stuff afterwards, and the history is now completely inaccessible to non-authenticated users, so there's no need to maintain my custom robots.txt Which is verifiably untrue: HTTP/1.1 200 server: nginx/1…
> intentional lie There are no “intentional” lies, because there are no “unintentional” lies. All lies are intentional. An “unintentional lie” is better known as “being wrong”. Being wrong isn’t always lying. What’s so hard about this? An example: My wife once asked me if I had taken the trash out to the curb, and I said I had. This was demonstrably false, anyone could see I had not. Yet for whatever reason, I mistak…