81–90 of 135 posts
In my opinion this is a transitional niche. Soon websites/apps whatever you want to call them will have their own built-in handling for AI. It's inefficient and rude to be scraping pages for content. Especially for profit.
I think each website/data-source having their own built-in AI is also a transitional period. It is like every website having search engine vs google.
I feel like this is unethical. You built yet another bot scraper. It would only be an ethical tool if it validated I own the website I am scraping before it starts.
Well, Google itself is just an unethical bot scrapper then...
This is not an easy problem to solve. In my naive take, authors get to decide how their work is used, not scrapers.
I like this a lot! But: I feel the more of these services come to being, the more likely it is that every website starts putting up gates to keep the bots away. Sort of like a weird GenAI take on Cixin Liu's Dark Forest hypothesis ( https://en.wikipedia.org/wiki/Dark_forest_hypothesis ). (Edited to add a reference.)
> I feel the more of these services come to being, the more likely it is that every website starts putting up gates to keep the bots away That's why we need microtransactions, because I'd rather be able to have both nice AI services and useful data repositories that they pull from, than have to choose just one. (and that one would be AI services, because you can't stop all the scrapers, so data sources will just keep…
I feel like this is unethical. You built yet another bot scraper. It would only be an ethical tool if it validated I own the website I am scraping before it starts.
This is probably a losing direction - protecting your little island of content in the sea of internet and LLM outputs. Get more value by exposure. This is the trend of open source, wikipedia and open scientific publication. LLMs double down on the same collaborative approach to intelligence. You can of course decouple from the big discussion and isolate your content with access restrictions, but the real interesting…
The LLM era doesn't give credit or attribution to its sources. It erases exposure. So there's a disincentive to collaborate with it, because it only takes.
> I think IP laws are in for a reckoning, protecting creativity by restricting it is not the best idea in the world.
We've been having this discussion for over 20 years since the Napster era, or even the era of elaborate anti piracy measures for computer games distributed on tapes 40 years ago.
I've reached the conclusion that the stable equilibrium is "small shadow world": enough IP leakage for piracy and preservation, but on a noncommercial scale. We sit with our Plex boxes and our adblockers, knowing that 90% of the world isn't doing that and is paying for it. Too much control is an IP monopoly stranglehold where it costs multiple dollars to set a song as your phone ringtone or briefly heard background music gets your video vaporised off social media. Too _little_ control and eventually there is actually a real economic loss from piracy, and original content does not get made.
AI presents a third threat: unlimited pseudo-creative "slop", which is cheap and adequate to fill people's scrolling time but does not pay humans for its creation and atrophys the creative ecosystem.
In my opinion this is a transitional niche. Soon websites/apps whatever you want to call them will have their own built-in handling for AI. It's inefficient and rude to be scraping pages for content. Especially for profit.
In agree that this niche is DOA. No offence to OP but the barrier for entry to this stuff is low. I built basically the same thing over a weekend for personal use. React frontend, python server, chroma for embeddings, sqlite cache, switch between open AI and anthropic (I want to add llama for full local execution when I get a better pc). I have a local SPA with named "projects", can configure crawl depth from a start…
Lmfao. God bless HN for keeping this meme going for decades by now.
Earlier quoted context omitted.
> I feel the more of these services come to being, the more likely it is that every website starts putting up gates to keep the bots away That's why we need microtransactions, because I'd rather be able to have both nice AI services and useful data repositories that they pull from, than have to choose just one. (and that one would be AI services, because you can't stop all the scrapers, so data sources will just keep…
We're never going to have microtransactions because of microfraud - and AI makes this problem worse rather than better.
Earlier quoted context omitted.
Well, Google itself is just an unethical bot scrapper then...
Several lawsuits have confirmed that. Google regurgitating articles from French newspaper sites comes to mind. This is not an easy problem to solve. In my naive take, authors get to decide how their work is used, not scrapers.
That was actually a big enlightening moment for me, as long as money is involved, the so called ethics were out of the window instantly. From the far left newspapers to the far right ones, they all lied on this topic. Only a handful tech blogs and newspapers did tell the truth.
I feel like this is unethical. You built yet another bot scraper. It would only be an ethical tool if it validated I own the website I am scraping before it starts.
yes only big conglomerates now can scrape pages. If you're not google stealing the info then...... Right?
In addition, this scraper doesn't even identify itself (I checked). It pretends to be a normal browser, without saying it's a scraper.
Earlier quoted context omitted.
We're never going to have microtransactions because of microfraud - and AI makes this problem worse rather than better.
What is this thing you've invented, "microfraud", and where's the evidence it exists?
Click fraud on adverts is a form of microfraud, and pay-per-click is the existing form of microtransaction.