Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Now AI agents need what RSS does

julienreszka.com

Post reply on HN
Print view View original

31–40 of 71 posts

Re: Now AI agents need what RSS does

#31

Post by Joe_Cool »

dchuk wrote: post #16

I built a site that's similar in concept to Hacker News, but is entirely fed by RSS feed content, that is then bullet-pointed summarized on the article page: https://engineered.at/ But I also extract topics automatically from the content too with LLMs, to allow for dynamic topic pages that users can separately subscribe to to tune their feeds. Haven't promoted it much, but it's pretty amazing what you can do for a co…

Getting

    406 browser not supported
for ESR Firefox 140.

If I set my UA to "FUCKIT" I can use the site perfectly fine. Why is there a User Agent Filter that disables the whole website? This should be maybe a warning, not a complete block.

Re: Now AI agents need what RSS does

#32

Post by Joe_Cool »

dchuk wrote: post #29
shaunpud wrote: post #25

Earlier quoted context omitted.

Your browser is not supported. Please upgrade your browser to continue. Can't even view your site with Firefox

That’s…bizarre. Let me take a look EDIT: just checked in firefox, I don't see an issue. can you email me at me@dchuk.com and maybe I can debug with you?

I just noticed the same thing.

UA being blocked for example:

  Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:140.0) Gecko/20100101 Firefox/140.0
Did mess with it some more:

Allowed:

    Opera/9.80 (Windows NT 6.1; U; zh-tw) Presto/2.7.62 Version/11.01
    Opera/9.80 (Windows NT 5.1; U; cs) Presto/2.7.62 Version/11.01
406:

    Mozilla/5.0 (Windows NT 5.1) Gecko/20100101 Firefox/14.0 Opera/12.0
    Mozilla/5.0 (Macintosh; Intel Mac OS X 14; rv:140.0) Gecko/20110101 Firefox/140.0
Maybe just remove it?

Re: Now AI agents need what RSS does

#33

Post by solid_fuel »

PaulHoule wrote: post #13

Re: Rate Limits, see https://rachelbythebay.com/w/2024/05/27/feed/ but coming from an aggressively anticommercial world view. She collects evidence that real world feed readers don't implement RSS correctly https://rachelbythebay.com/w/2026/02/23/readers/ Her problems are the problems of a polling-based protocol and really if she does not like the RSS protocol she should stop publishing it and stand up an ActivityPub…

> Her problems are the problems of a polling-based protocol and really if she does not like the RSS protocol she should stop publishing it and stand up an ActivityPub or PubSubHubBub service instead.

Bit odd to take potshots at a third party blog on this discussion, why single out Rachel?

And more to the point, the dynamics here might be due to RSS being polling-based, but if feed readers implemented the RSS logic correctly it wouldn't matter nearly as much, would it?

Re: Now AI agents need what RSS does

#34

Post by phyzix5761 »

sophiabits wrote: post #28
phyzix5761 wrote: post #18

Earlier quoted context omitted.

These are my top ones: Protesilaos: https://protesilaos.com/codelog.xml and https://protesilaos.com/commentary.xml HN: https://hnrss.org/frontpage Sacha Chua: https://sachachua.com/blog/feed/index.xml David Revoy: https://www.davidrevoy.com/feed/rss Davep: https://blog.davep.org/feeds/all.atom.xml xkcd: https://xkcd.com/atom.xml YouTube - Michelle Khare: https://www.youtube.com/feeds/videos.xml?channel_id=UCGGZ_PO...…

TIL you can get RSS feeds for YouTube channels. Thanks for this!

You're welcome. You can also turn any subreddit into a RSS feed: https://arkvis.com/blog/2026-04-16_interesting-finds.html

Re: Now AI agents need what RSS does

#36

Post by solid_fuel »

8organicbits wrote: post #19
b3ing wrote: post #7

I guess if you want your content all slurped up and served as coming from AI with no backlinks.

Anyone know the best practices for keeping AI crawlers off your RSS feeds? I know robots.txt works for the well-behaved bots. Other tools like interstitial captchas don't as the feed readers break if you send them anything but XML. Putting just the post intro in the feed and linking to the website feels like a safer approach, assume you have bot protections on the website, but that's a poor experience for people who…

I have some aggressive filters in Caddy that block the worst offenders by CIDR range, and also filter by user agent to remove any honest facebook and amazon bots. Otherwise, maybe strong rate limits by IP?

Edit:

Longer term, the approach might be - provide a separate RSS feed with full content but gated by a query parameter, then only give that URL to known-good consumers via email verification or patreon subscription, etc.

It would suck that people would have to pay more to consume content in their preferred way, but depending on your needs it might be a reasonable compromise.

Re: Now AI agents need what RSS does

#38

Post by _-_-__-_-_- »

frigidwalnut wrote: post #10
phyzix5761 wrote: post #9

I have almost 40 feeds I subscribe to and they're my primary way of getting information I care about without being exposed to ads or other things I don't want to see.

What are your favorite feeds?

https://theonlyblogever.com/blogroll opml file included!

Re: Now AI agents need what RSS does

#39

Post by dchuk »

Joe_Cool wrote: post #31
dchuk wrote: post #16

I built a site that's similar in concept to Hacker News, but is entirely fed by RSS feed content, that is then bullet-pointed summarized on the article page: https://engineered.at/ But I also extract topics automatically from the content too with LLMs, to allow for dynamic topic pages that users can separately subscribe to to tune their feeds. Haven't promoted it much, but it's pretty amazing what you can do for a co…

Getting 406 browser not supported for ESR Firefox 140. If I set my UA to "FUCKIT" I can use the site perfectly fine. Why is there a User Agent Filter that disables the whole website? This should be maybe a warning, not a complete block.

you know, I had setup some analytics filtering based on geoip because I was getting crazy spam traffic from Chine and Singapore, but that should only be affecting analytics not the whole site. Mind if I ask where you're located? (you can email me privately if preferred: me@dchuk.com)

Re: Now AI agents need what RSS does

#40

Post by devinpower »

dchuk wrote: post #16

I built a site that's similar in concept to Hacker News, but is entirely fed by RSS feed content, that is then bullet-pointed summarized on the article page: https://engineered.at/ But I also extract topics automatically from the content too with LLMs, to allow for dynamic topic pages that users can separately subscribe to to tune their feeds. Haven't promoted it much, but it's pretty amazing what you can do for a co…

This looks great, I've wanted something like this for a while. Finding how to click through to the actual item in the feed was a high point of friction for me.

I went to a topic and then clicked on the header of something I was interested in expecting to be brought to the blog post directly. Needing to click on that same title again to be brought to the post was unintuitive to me, I searched around the page, went back and forth a few times and eventually figured it out.

As a user I would love to be able to click directly through to the article FROM the topic feed. I would expect that the comments is a URL to the page that the header currently brings me to. This would match my expectations from using sites like reddit/HN.

A one or two liner summary directly on the topics feed would be really great I think.

Post reply on HN