Awesome, I once developed a project called https://rerss.xyz, aimed at creating an RSS feed that reorders historical blog posts, but it was hindered by the two issues mentioned in the article.
Show HN: Full Text, Full Archive RSS Feeds for Any Blog
41–50 of 54 posts
Re: Show HN: Full Text, Full Archive RSS Feeds for Any Blog
#42Someone somewhere is still running a gopher server.
Re: Show HN: Full Text, Full Archive RSS Feeds for Any Blog
#43Earlier quoted context omitted.
It's not what you're aiming for with this comment, but I bet git would actually make a pretty good storage tool/format for archival of mostly static sites. horrible simple hack: use `wget` with `--mirror` option, and commit the result to a git repository. Repeat with a `cron` job to keep an archive with change history.
I assume this is what wayback machine uses?
Of course not. They have their own crawler (Heritrix, an open source Java crawler) and archive in WARC format. It‘s serious archiving, they want to preserve reply codes, HTTP headers etc.
Re: Show HN: Full Text, Full Archive RSS Feeds for Any Blog
#44As a publisher who publishes a full-text RSS feed at a time when not a lot of publishers do, I must say: The publisher should have a say in this. This is not to say that this is a good idea or a bad one, but I think you will, long-term, have better luck if people don’t feel their content is being siphoned. A great case-in-point is what my friends at 404 Media did: https://www.404media.co/why-404-media-needs-your-emai…
I disagree. If you put your content out in the open for everyone to read, it is totally valid to scrape that content. Otherwise put it behind a paywall. If i can access it for free with a browser then you should be fine with me consuming your content with the tool of my choice. So i can search or use it however i see fit. Why not?
Getting consumed by ai scrapers will be inevitable in the long run i think.
Re: Show HN: Full Text, Full Archive RSS Feeds for Any Blog
#45As a publisher who publishes a full-text RSS feed at a time when not a lot of publishers do, I must say: The publisher should have a say in this. This is not to say that this is a good idea or a bad one, but I think you will, long-term, have better luck if people don’t feel their content is being siphoned. A great case-in-point is what my friends at 404 Media did: https://www.404media.co/why-404-media-needs-your-emai…
I disagree. If you put your content out in the open for everyone to read, it is totally valid to scrape that content. Otherwise put it behind a paywall. If i can access it for free with a browser then you should be fine with me consuming your content with the tool of my choice. So i can search or use it however i see fit. Why not? Getting consumed by ai scrapers will be inevitable in the long run i think.
As a one-off thing you personally do, yeah that’s probably okay. Turning that into a product that you then offer to others is where the line is drawn, in my opinion.
Re: Show HN: Full Text, Full Archive RSS Feeds for Any Blog
#46As a publisher who publishes a full-text RSS feed at a time when not a lot of publishers do, I must say: The publisher should have a say in this. This is not to say that this is a good idea or a bad one, but I think you will, long-term, have better luck if people don’t feel their content is being siphoned. A great case-in-point is what my friends at 404 Media did: https://www.404media.co/why-404-media-needs-your-emai…
I disagree. If you put your content out in the open for everyone to read, it is totally valid to scrape that content. Otherwise put it behind a paywall. If i can access it for free with a browser then you should be fine with me consuming your content with the tool of my choice. So i can search or use it however i see fit. Why not? Getting consumed by ai scrapers will be inevitable in the long run i think.
I feel like the two massive unspoken caveats are:
1. Downloading and polling that doesn't resemble a cyberattack.
2. Not reproducing their content in a way that could compete with theirs or tarnishes their identity... and there's a lot of open ongoing debate about how that principle relates to different ways of using LLMs.
Re: Show HN: Full Text, Full Archive RSS Feeds for Any Blog
#47As a publisher who publishes a full-text RSS feed at a time when not a lot of publishers do, I must say: The publisher should have a say in this. This is not to say that this is a good idea or a bad one, but I think you will, long-term, have better luck if people don’t feel their content is being siphoned. A great case-in-point is what my friends at 404 Media did: https://www.404media.co/why-404-media-needs-your-emai…
I disagree. If you put your content out in the open for everyone to read, it is totally valid to scrape that content. Otherwise put it behind a paywall. If i can access it for free with a browser then you should be fine with me consuming your content with the tool of my choice. So i can search or use it however i see fit. Why not? Getting consumed by ai scrapers will be inevitable in the long run i think.
Just because I make the information available in a convenient way doesn't mean I expect it to be harvested. That you make that leap is 100% troubling and makes me not want to have you as a reader, because you don't respect my work.
You are describing the “give an inch, take a mile” concept neatly.
I think your mindset will just lead to a lot of people who otherwise would not want to regwall their content to do so. And if I ever do so, I will include a link to your post so they know who to blame.
Re: Show HN: Full Text, Full Archive RSS Feeds for Any Blog
#48Earlier quoted context omitted.
I disagree. If you put your content out in the open for everyone to read, it is totally valid to scrape that content. Otherwise put it behind a paywall. If i can access it for free with a browser then you should be fine with me consuming your content with the tool of my choice. So i can search or use it however i see fit. Why not? Getting consumed by ai scrapers will be inevitable in the long run i think.
As a one-off thing you personally do, yeah that’s probably okay. Turning that into a product that you then offer to others is where the line is drawn, in my opinion.
I think this is a fair line. I don't want to mess with the tinkerers of the world, and to be clear I'm not even entirely opposed to this. I just think we do not put enough stock into discussing potentially damaging actions with creators.
Which is why so many writers and artists are upset at OpenAI and Anthropic right now.
Re: Show HN: Full Text, Full Archive RSS Feeds for Any Blog
#49As a publisher who publishes a full-text RSS feed at a time when not a lot of publishers do, I must say: The publisher should have a say in this. This is not to say that this is a good idea or a bad one, but I think you will, long-term, have better luck if people don’t feel their content is being siphoned. A great case-in-point is what my friends at 404 Media did: https://www.404media.co/why-404-media-needs-your-emai…
ive never implemented it but it should be possible to check if content still lives behind the url where it was originally found before serving any kind of archived copy.(preferably with contact info for the unwilling author) Using it for a search index should be fine ofc
Re: Show HN: Full Text, Full Archive RSS Feeds for Any Blog
#50Someone somewhere is still running a gopher server.
Having a fixed path for a search api is a great idea.