Live data from Hacker News

Deep dive into finding RSS feeds

lighthouseapp.io

21–30 of 65 posts

Re: Deep dive into finding RSS feeds

#21
It would be nice if someone ran this on commoncrawl and published a list of all the RSS feeds. (probably someone has?)

Or I suppose you could just find all "Content-type: application/rss+xml" in CC.

I know in the past, when I was looking for large lists of RSS feeds, I didn't really find what I was looking for.

Re: Deep dive into finding RSS feeds

#23

Back when I was young, websites had this icon you could click that would take you straight to their RSS feed. You young whipper snappers have gone an fucked that up. Actually, I think it was Google's fault. When they killed their RSS reader people pronounced RSS dead so people just stopped publishing RSS feeds or just didn't link to them. * Yes, I know the article talks about the RSS icon, i'm just soapboxing.

Even better, for a few months the browsers themselves would highlight RSS feeds and allow you subscribe right in the browser. It was too good to last. RSS is great but it has one great flaw in that it doesn't scale that well by itself. If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RSS. What does work well (and how things have evolved) is to ha…

> RSS is great but it has one great flaw in that it doesn't scale that well by itself. If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RSS.

Who's updating their RSS feeds once an hour 24 hours a day?

Anyway, a very popular site with that many millions of visitors already has to handle extreme traffic, regardless of RSS.

Re: Deep dive into finding RSS feeds

#25

Back when I was young, websites had this icon you could click that would take you straight to their RSS feed. You young whipper snappers have gone an fucked that up. Actually, I think it was Google's fault. When they killed their RSS reader people pronounced RSS dead so people just stopped publishing RSS feeds or just didn't link to them. * Yes, I know the article talks about the RSS icon, i'm just soapboxing.

Even better, for a few months the browsers themselves would highlight RSS feeds and allow you subscribe right in the browser. It was too good to last. RSS is great but it has one great flaw in that it doesn't scale that well by itself. If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RSS. What does work well (and how things have evolved) is to ha…

> Even better, for a few months the browsers themselves would highlight RSS feeds and allow you subscribe right in the browser. It was too good to last.

I use the Awesome RSS add-on for that in Firefox: https://addons.mozilla.org/en-US/firefox/addon/awesome-rss/

Re: Deep dive into finding RSS feeds

#26
post #23

Earlier quoted context omitted.

Even better, for a few months the browsers themselves would highlight RSS feeds and allow you subscribe right in the browser. It was too good to last. RSS is great but it has one great flaw in that it doesn't scale that well by itself. If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RSS. What does work well (and how things have evolved) is to ha…

> RSS is great but it has one great flaw in that it doesn't scale that well by itself. If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RSS. Who's updating their RSS feeds once an hour 24 hours a day? Anyway, a very popular site with that many millions of visitors already has to handle extreme traffic, regardless of RSS.

Surely this is solved by putting RSS behind Cloudflare.

Re: Deep dive into finding RSS feeds

#27
post #23

Earlier quoted context omitted.

Even better, for a few months the browsers themselves would highlight RSS feeds and allow you subscribe right in the browser. It was too good to last. RSS is great but it has one great flaw in that it doesn't scale that well by itself. If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RSS. What does work well (and how things have evolved) is to ha…

> RSS is great but it has one great flaw in that it doesn't scale that well by itself. If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RSS. Who's updating their RSS feeds once an hour 24 hours a day? Anyway, a very popular site with that many millions of visitors already has to handle extreme traffic, regardless of RSS.

According to my logs, some feed readers poll every 10 minutes or so.

Re: Deep dive into finding RSS feeds

#28
post #18

Earlier quoted context omitted.

Even better, for a few months the browsers themselves would highlight RSS feeds and allow you subscribe right in the browser. It was too good to last. RSS is great but it has one great flaw in that it doesn't scale that well by itself. If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RSS. What does work well (and how things have evolved) is to ha…

> RSS is great but it has one great flaw in that it doesn't scale that well by itself The readers often cache the result for all users of the same feed. The only exception are readers that are running 100% locally. In that case using Etag or Last-Modified will make each request cheap and manageble.

That is my point. RSS by itself is a scaling problem but online readers make it manageable by polling sites on a reasonable rate and redistributing new articles to their users.

Re: Deep dive into finding RSS feeds

#30
post #23

Earlier quoted context omitted.

> RSS is great but it has one great flaw in that it doesn't scale that well by itself. If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RSS. Who's updating their RSS feeds once an hour 24 hours a day? Anyway, a very popular site with that many millions of visitors already has to handle extreme traffic, regardless of RSS.

Surely this is solved by putting RSS behind Cloudflare.

Sure, it's solved by Cloudflare breaking the RSS feed entirely, which often happens when people put their websites behind Cloudflare.
Post reply on HN