181–190 of 379 posts
Wow, reddit found a way to make themselves even less useful somehow. After the API fiasco, that seemed like it'd be pretty hard to do.
The API changes and these robots.txt were part of the same strategy - preventing third parties from scrapping their data and reducing the AI generated content that makes it into their data. So they can sell that data and make money.
Earlier quoted context omitted.
Bing provides far fewer verbatim results for pretty much all search queries that I've tested. And Yandex isn't much better for non cyrillic search, Baidu is only for the Chinese web effectively. And all other search engines either don't even attempt to do full web crawls anymore and/or buy from one of the four above. So realistically there's just one search engine for the full web that actually does the work.
> And Yandex isn't much better for non cyrillic search, I like Yandex when I'm rabbit-holing after obscure musicians/music. I routinely have a better experience than I do with DDG or Kagi or Goog.
Earlier quoted context omitted.
FYI, Kagi lets you do this and personalize it as you desire. They even share aggregated stats※ about which domains users choose to block/lower. (Mine generally match these stats.) ※ - https://kagi.com/stats?stat=leaderboard&k=-2
> Kagi lets you do this and personalize it as you desire. Kagi shill here. Are they finally applying filters and operands to image searches? Asking because it was a tough year seeing Pinterest as top filter choice and top result in images (when set as filter=block). (edit: I just tried searching->image: beautiful quilt patterns. I didn't spot any Pinterest results!) I have never understood why DDG, etc steadfastly re…
That was a bug, apologies. It should be fixed now.
site:reddit.com works for kagi for new posts this week?
Kagi gets part of their index from Google, per the article, so perhaps that's the reason Kagi still works. Wonder if Vlad and Kagi will do (or have done) the calculus to see if buying crawlability from Reddit itself is cheaper than buying results from Google for Reddit search.
Wow, reddit found a way to make themselves even less useful somehow. After the API fiasco, that seemed like it'd be pretty hard to do.
The API changes and these robots.txt were part of the same strategy - preventing third parties from scrapping their data and reducing the AI generated content that makes it into their data. So they can sell that data and make money.
Love how it's their data when it might make them money but not their data if they get sued.
This is a dangerous precedent for the internet. Business conglomerates have been controlling most of the web, but refusing basic interoperability is even worse.
Earlier quoted context omitted.
reddit blocked datacenter IPs even before this change.
Could a motivated scraper not buy IPs/proxies that aren’t in those ranges, i.e. to blend in with general users?
Eventually it becomes expensive to scrape reddit's data and most people will stop.
When Microsoft strikes an exclusive deal with OpenAI to use their models, it is a smart, brilliant, clever move. When Apple strikes an exclusive deal with suppliers for parts, it is sound business practice. When Google strikes an exclusive deal with Reddit, it is .. Some of you have no idea how businesses work, and it shows.
It's because reddit is selling content created by users, base on promises that reddit supports open internet, open data, etc, without their consent and sharing revenue, which maybe legal but likely not ethical.
Worse it doesn't even really "work" anymore, giving how most search are flooded with garbage SEO results and payed advertisements "basically" looking like search results (most times more garbage not what you are looking for results, int he cases where it isn't it quite often times is on the line of "googles algorithm blackmailing companies to buy ads for users which want to find them through google but wouldn't witho…
> Worse it doesn't even really "work" anymore, giving how most search are flooded with garbage SEO results and payed advertisements "basically" looking like search results ... I don't understand what you're saying. That's exactly why people append `site:reddit.com` to their searches in the first place, because those search results typically aren't like that.
They changed robots.txt a month or so ago. For the first 19 years of life, reddit had a very permissive robots.txt. We allowed all by default and then only restricted certain poorly behaved agents (and Bender's Shiny Metal Ass(tm)) But I can understand why they made the change they did. The data was being abused. My guess is that this was an oversight -- that they will do an audit and reopen it for search engines aft…
Person extensively quoted in the article here. They are welcome to reach out. But not a single person from any level did that, nor replied to my polite requests to explain and engage. We first contacted them in early June and by 13th June, I had escalated to Steve Huffman @spez.
Shared because it is unlikely Reddit responds except when required by law, so I recommend engaging regulators (FTC, and DOJ at the bare minimum) and legislators (primarily those focused on Section 230 reforms) whenever possible with regards to this entity. They're the only folks worth escalating to, as Reddit's incentives are to gate content, keep ad buyers happy, and keep the user base in check while they struggle to break even, sharing as little information publicly as possible along the way [1] [2].
[1] https://www.bloomberg.com/news/articles/2024-05-09/reddit-la... | https://archive.today/wQuKM