Deep dive into finding RSS feeds • Hacker News Board

Re: Deep dive into finding RSS feeds

#31

Post by maxglute » Sat, Dec 07, 2024, 1:46 AM UTC

HumblyTossed wrote: post #6

Back when I was young, websites had this icon you could click that would take you straight to their RSS feed. You young whipper snappers have gone an fucked that up. Actually, I think it was Google's fault. When they killed their RSS reader people pronounced RSS dead so people just stopped publishing RSS feeds or just didn't link to them. * Yes, I know the article talks about the RSS icon, i'm just soapboxing.

Feedly has relatively adequate RSS builder based on site elements for sites without RSS, but you only get a few feeds with pro plan. Wish there was open implementation. Also seems like something LLM would excel at.

Re: Deep dive into finding RSS feeds

#32

Post by doubled112 » Sat, Dec 07, 2024, 1:49 AM UTC

lapcat wrote: post #23

AndrewStephens wrote: post #9

Earlier quoted context omitted.

Even better, for a few months the browsers themselves would highlight RSS feeds and allow you subscribe right in the browser. It was too good to last. RSS is great but it has one great flaw in that it doesn't scale that well by itself. If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RSS. What does work well (and how things have evolved) is to ha…

> RSS is great but it has one great flaw in that it doesn't scale that well by itself. If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RSS. Who's updating their RSS feeds once an hour 24 hours a day? Anyway, a very popular site with that many millions of visitors already has to handle extreme traffic, regardless of RSS.

I think Miniflux polls every hour by default. Not sure about others.

Re: Deep dive into finding RSS feeds

#33

Post by panozzaj » Sat, Dec 07, 2024, 2:14 AM UTC

I use a Chrome extension (https://chromewebstore.google.com/detail/get-rss-feed-url/kf...) and it seems to pick out the RSS URLs fairly consistently

Re: Deep dive into finding RSS feeds

#34

Post by ndriscoll » Sat, Dec 07, 2024, 4:11 AM UTC

AndrewStephens wrote: post #9

HumblyTossed wrote: post #6

Back when I was young, websites had this icon you could click that would take you straight to their RSS feed. You young whipper snappers have gone an fucked that up. Actually, I think it was Google's fault. When they killed their RSS reader people pronounced RSS dead so people just stopped publishing RSS feeds or just didn't link to them. * Yes, I know the article talks about the RSS icon, i'm just soapboxing.

Even better, for a few months the browsers themselves would highlight RSS feeds and allow you subscribe right in the browser. It was too good to last. RSS is great but it has one great flaw in that it doesn't scale that well by itself. If 2 million people subscribe to your feed and try to update it once an hour, that is 48 million requests a day just for RSS. What does work well (and how things have evolved) is to ha…

A $160 n100 minipc should have no problem serving an essentially static file out of nginx at well above that rate. If you use proper caching headers and don't publish new items that often, it should basically be idle.

Re: Deep dive into finding RSS feeds

#35

Post by fallinditch » Sat, Dec 07, 2024, 4:41 AM UTC

This looks very useful. It would work well with Hoarder (would be cool if they were integrated ;)

Note: Hoarder can automatically hoard RSS feeds as part of its 'bookmark everything' functionality. Hoarder uses AI to tag all the content (URLs, feeds, images, notes) so you can then do full text searches on your personal archive of your bookmarks etc.

https://hoarder.app/

Re: Deep dive into finding RSS feeds

#36

Post by freetonik » Sat, Dec 07, 2024, 5:33 AM UTC

ks2048 wrote: post #21

It would be nice if someone ran this on commoncrawl and published a list of all the RSS feeds. (probably someone has?) Or I suppose you could just find all "Content-type: application/rss+xml" in CC. I know in the past, when I was looking for large lists of RSS feeds, I didn't really find what I was looking for.

Far from a complete list, but I maintain a curated list of personal blogs with RSS feeds here https://minifeed.net/blogs

Re: Deep dive into finding RSS feeds

#37

Post by compootr » Sat, Dec 07, 2024, 7:12 AM UTC

lapcat wrote: post #30

ttyprintk wrote: post #26

Earlier quoted context omitted.

Surely this is solved by putting RSS behind Cloudflare.

Sure, it's solved by Cloudflare breaking the RSS feed entirely, which often happens when people put their websites behind Cloudflare.

I think it should be noted that this happens when not configuring things right.

By default, security rules are on, but I can disable security rules programmatically for a hostname too.

fwiw, I once got a ddos on a host running their pages product, and got no charge for it. it also stayed up and didn't give captcha pages

Re: Deep dive into finding RSS feeds

#38

Post by PeterStuer » Sat, Dec 07, 2024, 7:55 AM UTC

You can add '/display-feed.rss' to the list of common suffixes for many .eu sites

Re: Deep dive into finding RSS feeds

#39

Post by djhn » Sat, Dec 07, 2024, 8:13 AM UTC

fallinditch wrote: post #35

This looks very useful. It would work well with Hoarder (would be cool if they were integrated ;) Note: Hoarder can automatically hoard RSS feeds as part of its 'bookmark everything' functionality. Hoarder uses AI to tag all the content (URLs, feeds, images, notes) so you can then do full text searches on your personal archive of your bookmarks etc. https://hoarder.app/

Awesome piece of software and right up my alley (although I’m too invested in my homegrown scripts to let them go - but I’ll give this a try.)

You might consider adding some of the key info from the first paragraphs in the docs (open source, self hosting) to the front page, above the fold. Github-link might imply it, but I was scanning for “open source” with my eyes and was initially disappointed not to find it and ready to dismiss the product right out of the gate.

Re: Deep dive into finding RSS feeds

#40

Post by aucisson_masque » Sat, Dec 07, 2024, 8:40 AM UTC

I thank WordPress for most of my RSS feed.

I follow mostly RSS on non technology website, for instance road cycling. people that wouldn't care or know about RSS because they are not very techy, yet because they are normies that use WordPress for all their website it puts a page with RSS feed automatically. You got to find it with developer tool by searching RSS but 99% of the time if it's WordPress it got RSS.

Thank you WordPress you bloated piece of shit :)