Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep dive into finding RSS feeds

lighthouseapp.io

Post reply on HN
Print view View original

51–60 of 65 posts

Re: Deep dive into finding RSS feeds

#51

Post by lapcat »

xtiansimon wrote: post #50
lapcat wrote: post #23

Earlier quoted context omitted.

> RSS is great but it has one great flaw in that it doesn't scale that well by itself. If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RSS. Who's updating their RSS feeds once an hour 24 hours a day? Anyway, a very popular site with that many millions of visitors already has to handle extreme traffic, regardless of RSS.

“Who's updating their RSS feeds once an hour 24 hours a day?” RSS is a pull-type system, no?. So the end-user is causing the overload problem by hitting the publisher’s RSS feed every hour? The problem is out of the hands of the publisher…

> RSS is a pull-type system, no?

Yes.

The web in general is also a pull-type system.

> So the end-user is causing the overload problem by hitting the publisher’s RSS feed every hour?

There is no overload problem.

And again, this math is off, because an end user is not even awake 24 hours a day: "If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RSS."

Re: Deep dive into finding RSS feeds

#52

Post by doubled112 »

lapcat wrote: post #51
xtiansimon wrote: post #50

Earlier quoted context omitted.

“Who's updating their RSS feeds once an hour 24 hours a day?” RSS is a pull-type system, no?. So the end-user is causing the overload problem by hitting the publisher’s RSS feed every hour? The problem is out of the hands of the publisher…

> RSS is a pull-type system, no? Yes. The web in general is also a pull-type system. > So the end-user is causing the overload problem by hitting the publisher’s RSS feed every hour? There is no overload problem. And again, this math is off, because an end user is not even awake 24 hours a day: "If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RS…

My home server is awake 24 hours a day.

Is that wasteful? Does that make me a poor Internet citizen?

I'd never considered using RSS feeds from an app and pulling it directly. I have too many devices for that to be a great workflow.

Normally I'd say I was an outlier, but my gut feeling on this is that the people still using RSS and people self-hosting would overlap in a big way.

Re: Deep dive into finding RSS feeds

#53

Post by fallinditch »

djhn wrote: post #39
fallinditch wrote: post #35

This looks very useful. It would work well with Hoarder (would be cool if they were integrated ;) Note: Hoarder can automatically hoard RSS feeds as part of its 'bookmark everything' functionality. Hoarder uses AI to tag all the content (URLs, feeds, images, notes) so you can then do full text searches on your personal archive of your bookmarks etc. https://hoarder.app/

Awesome piece of software and right up my alley (although I’m too invested in my homegrown scripts to let them go - but I’ll give this a try.) You might consider adding some of the key info from the first paragraphs in the docs (open source, self hosting) to the front page, above the fold. Github-link might imply it, but I was scanning for “open source” with my eyes and was initially disappointed not to find it and r…

I didn't make this software, but heard about it on another thread. I was saying 'there should be an app like an 'auto-RAG' that scrapes RSS feeds and URLs' and be_erik wrote 'I do exactly this with hoarder. I passively build tagged knowledge bases with the archived pages and then feed it to a RAG setup.'

What does your setup look like?

Re: Deep dive into finding RSS feeds

#54

Post by lapcat »

doubled112 wrote: post #52
lapcat wrote: post #51

Earlier quoted context omitted.

> RSS is a pull-type system, no? Yes. The web in general is also a pull-type system. > So the end-user is causing the overload problem by hitting the publisher’s RSS feed every hour? There is no overload problem. And again, this math is off, because an end user is not even awake 24 hours a day: "If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RS…

My home server is awake 24 hours a day. Is that wasteful? Does that make me a poor Internet citizen? I'd never considered using RSS feeds from an app and pulling it directly. I have too many devices for that to be a great workflow. Normally I'd say I was an outlier, but my gut feeling on this is that the people still using RSS and people self-hosting would overlap in a big way.

> Is that wasteful? Does that make me a poor Internet citizen?

I personally don't think it's a problem. As I said, "a very popular site with that many millions of visitors already has to handle extreme traffic, regardless of RSS."

> I'd never considered using RSS feeds from an app and pulling it directly. I have too many devices for that to be a great workflow.

A lot of apps have sync to handle that workflow.

> my gut feeling on this is that the people still using RSS and people self-hosting would overlap in a big way.

My gut feeling is that most RSS users, including myself, simply use client apps.

Re: Deep dive into finding RSS feeds

#55

Post by DanAtC »

ndriscoll wrote: post #34
AndrewStephens wrote: post #9

Earlier quoted context omitted.

Even better, for a few months the browsers themselves would highlight RSS feeds and allow you subscribe right in the browser. It was too good to last. RSS is great but it has one great flaw in that it doesn't scale that well by itself. If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RSS. What does work well (and how things have evolved) is to ha…

A $160 n100 minipc should have no problem serving an essentially static file out of nginx at well above that rate. If you use proper caching headers and don't publish new items that often, it should basically be idle.

A lot of clients don't respect caching headers[0], but yeah, static content hosting is a solved problem.

And there are always CDNs if you really have a large, global audience.

[0] http://rachelbythebay.com/w/2024/10/25/fs/

Re: Deep dive into finding RSS feeds

#56

Post by sitkack »

monooso wrote: post #46
sitkack wrote: post #45

Earlier quoted context omitted.

Firefox removing RSS support was their capitulation in supporting the open web.

That is very ambiguous phrasing.

Firefox removing RSS support was the nail in the coffin for their support of the Open Web.

How long after Google Reader was canceled did FF remove RSS?

Re: Deep dive into finding RSS feeds

#57

Post by renegat0x0 »

I fought this problem, since I wrote my own RSS reader in python. Might not be perfect.

The problem with the approach presented here is speed. Most of the web pages, especially smaller are really slow.

Crawling most of the web pages is pain, especially if you use selenium and small SBC.

Therefore either the page presents a clean nice RSS link, or get lost.

Most of the good, modern pages give you nice RSS. Even GitHub gives you RSS for commits.

For other pages I try openRSS.

For YouTube I use yt-dlp to obtain channel id, to establish RSS.

Algorithm is crude, but gets the job done.

https://github.com/rumca-js/Django-link-archive/blob/main/rs...

Re: Deep dive into finding RSS feeds

#58

Post by djhn »

fallinditch wrote: post #53
djhn wrote: post #39

Earlier quoted context omitted.

Awesome piece of software and right up my alley (although I’m too invested in my homegrown scripts to let them go - but I’ll give this a try.) You might consider adding some of the key info from the first paragraphs in the docs (open source, self hosting) to the front page, above the fold. Github-link might imply it, but I was scanning for “open source” with my eyes and was initially disappointed not to find it and r…

I didn't make this software, but heard about it on another thread. I was saying 'there should be an app like an 'auto-RAG' that scrapes RSS feeds and URLs' and be_erik wrote 'I do exactly this with hoarder. I passively build tagged knowledge bases with the archived pages and then feed it to a RAG setup.' What does your setup look like?

Things I bookmark (browsers), like and save (youtube, tiktok), upvote (hn) get archived to both internet archive and my own storage.

Re: Deep dive into finding RSS feeds

#59

Post by rakoo »

AndrewStephens wrote: post #9
HumblyTossed wrote: post #6

Back when I was young, websites had this icon you could click that would take you straight to their RSS feed. You young whipper snappers have gone an fucked that up. Actually, I think it was Google's fault. When they killed their RSS reader people pronounced RSS dead so people just stopped publishing RSS feeds or just didn't link to them. * Yes, I know the article talks about the RSS icon, i'm just soapboxing.

Even better, for a few months the browsers themselves would highlight RSS feeds and allow you subscribe right in the browser. It was too good to last. RSS is great but it has one great flaw in that it doesn't scale that well by itself. If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RSS. What does work well (and how things have evolved) is to ha…

That's exactly why pubsubhubbub, now websub, has been created. Around 10 years ago.
Post reply on HN