Show HN: Full Text, Full Archive RSS Feeds for Any Blog

Before building an app that depends on the Wayback Machine (or other Archive infrastructure) it's good to keep in mind this post from their blog: https://blog.archive.org/2023/05/29/let-us-serve-you-but-don...>

One of my favorite tricks when coming across a blog with a longtail of past posts is to verify that it's hosted on WordPress and then to ingest the archives into my feedreader.

Once you have the WordPress feed URL, you can slurp it all in by appending `?paged=n` (or `&paged=n`) for the nth page of the feed. (This is a little tedious in Thunderbird; up till now I've generated a list of URLs and dragged and dropped each one into the subscribe-to-feed dialog. The whole process is amenable to scripting by bookmarklet, though—gesture at a blog with the appropriate metadata, and then get a file that's one big RSS/Atom container with every blog post.)

Re: Show HN: Full Text, Full Archive RSS Feeds for Any Blog

#7

Post by latexr » Mon, Sep 02, 2024, 2:44 PM UTC

> RSS and ATOM feeds are problematic for two reasons; 1) lack of history, 2) contain limited post content.

None of those are problems with RSS or Atom¹ feeds. There’s no technical limitation to having the full history and full post content in the feeds. Many feeds behave that way due to a choice by the author or as the default behaviour of the blogging platform. Both have reasons to be: saving bandwidth² and driving traffic to the site³.

Which is not to say what you just made doesn’t have value. It does, and kudos for making it. But twice at the top of your post you’re making it sound as if those are problems inherit with the format when they’re not. They’re not even problems for most people in most situations, you just bumped into a very specific use-case.

¹ It’s not an acronym, it shouldn’t be all uppercase.

² Many feed readers misbehave and download the whole thing instead of checking ETags.

³ To show ads or something else.

Re: Show HN: Full Text, Full Archive RSS Feeds for Any Blog

#8

Post by steamodon » Mon, Sep 02, 2024, 2:49 PM UTC

I wrote a similar tool [1], although it's designed to let you gradually catch up on a backlog rather than write a full feed all at once. Right now it only works on Blogger and WordPress blogs, so I'll need to learn from their trick of pulling from Internet Archive.

[1] https://github.com/steadmon/blog-replay

Re: Show HN: Full Text, Full Archive RSS Feeds for Any Blog

#9

Post by z3t4 » Mon, Sep 02, 2024, 3:00 PM UTC

The mystical creature - the URL - is a link to a resource that doesn't have to be static, it's only the URL that is static. eg. the content might change. So you might want to have the program revisit the resource once in a while to see if there are updates.

Re: Show HN: Full Text, Full Archive RSS Feeds for Any Blog

#10

Post by simonw » Mon, Sep 02, 2024, 3:13 PM UTC

yawnxyz wrote: post #4

It's so clever to just pull from Wayback Machine rather than scrape the site itself. Never even thought of that

I used it to recover some lost content from my blog a few years ago, it was fantastic: https://simonwillison.net/2017/Oct/8/missing-content/