I have this idea, that instead of browsing completely random things on the internet pushed by what other people are interested in (or want to promote), create an llm that scans through your backlog of projects YOU want to do, and then search the internet for projects/articles about those things, and then create a feed from that. I'm not sure why I keep reading HN, 99% of the content is uninteresting, probably 99.9% n…
This is going to happen, but it's too expensive for your LLM to do the scanning, and instead someone needs to build and maintain the index while allowing other people to subscribe to concepts. The problem is no one has sorted out the embedding space this all lives in.
Now AI agents need what RSS does
61–70 of 71 posts
Re: Now AI agents need what RSS does
#62I have this idea, that instead of browsing completely random things on the internet pushed by what other people are interested in (or want to promote), create an llm that scans through your backlog of projects YOU want to do, and then search the internet for projects/articles about those things, and then create a feed from that. I'm not sure why I keep reading HN, 99% of the content is uninteresting, probably 99.9% n…
This is going to happen, but it's too expensive for your LLM to do the scanning, and instead someone needs to build and maintain the index while allowing other people to subscribe to concepts. The problem is no one has sorted out the embedding space this all lives in.
I'd argue LLMs are getting cheaper, so it will get more feasible for LLMs to soon act on our behalf, bringing only what we're interested in.
Projects like OpenClaw and Hermes already show that this can work whether the source is RSS or simply a website the agent visits.
Even Google now envisions this, since they recently announced "information agents" (https://blog.google/products-and-platforms/products/search/s...) that will keep working in the background. They surely have an index they can use, but I wonder whether that's necessary? AI agents like Claude Code suggest it's possible to use simple keyword searches, without maintaining vector indexes - https://www.tigerdata.com/blog/why-cursor-is-about-to-ditch-...
It could be that soon we're gonna get a fully personalized briefing on the topics that we're interested in, or maybe a new kind of feed, replacing social media.
I'm actually working on the briefing idea myself: https://briefin.com
Re: Now AI agents need what RSS does
#63The issue with RSS is that it doesn't allow to show ads. That is why the RSS reader was killed by Google.
Nowadays AI agents also don't read ads. Let's see how that is going, but the ad industry isn't amused about that.
Re: Now AI agents need what RSS does
#64I have this idea, that instead of browsing completely random things on the internet pushed by what other people are interested in (or want to promote), create an llm that scans through your backlog of projects YOU want to do, and then search the internet for projects/articles about those things, and then create a feed from that. I'm not sure why I keep reading HN, 99% of the content is uninteresting, probably 99.9% n…
This is going to happen, but it's too expensive for your LLM to do the scanning, and instead someone needs to build and maintain the index while allowing other people to subscribe to concepts. The problem is no one has sorted out the embedding space this all lives in.
That's why I reached for Apples own local LLM to fool with similar ideas like this: https://pageforth.com. Apple is better than I expected at this. Right now it filters through things like hacker news articles and whatever else you point it at to summarize and find things that match your interests. Apple's LLM reminds me of Claude like 3 years ago. It's weak for sure. But useful for small dose kind of problems.
Re: Now AI agents need what RSS does
#65Agree on RSS as the right shape — and worth adding the cost angle nobody's
quantified here yet. Having an LLM read a 50KB HTML page is ~$0.03 of
gpt-4o input. Polling 1000 sources hourly = ~$720/day, almost all of it
tokenizing layout chrome the model throws away. RSS-shaped feeds drop
that 90%+ because they strip to deltas. The harder blocker is the supply
side though — publishers earn pennies per human pageview from ads and ~$0
from agent polls, so unless feeds become licensed paid endpoints, the
publisher incentive runs against your "publish an RSS feed for your
content" recommendation. Just like that :)
Re: Now AI agents need what RSS does
#66I kinda don't like RSS because I often want like a whole blog archive downloaded if I add a new feed and it usually has limits how far back of posts it will download (randomly configured by each site) Unless someone has a fix of whatever settings I've been using
I had a similar issue where I wanted to read a newly-followed blog from the beginning instead of the point in time when I started to follow, so I created https://refeed.to/
Re: Now AI agents need what RSS does
#67Earlier quoted context omitted.
As a sysadmin hosting a few blogs, do you mind sharing what IP ranges you crawl from? Or what agent your requests use? Thank you.
I presume you’re politely asking in order to block? Which is fine, I get it. On my phone right now but can update later. I do want to ask though (and I should make this clear in a FAQ or something): the way I check RSS feeds uses adaptive scheduling, so I intentionally don’t check feeds of sites too rapidly. Then the summarization is based on the full article content but I never render that full content on the site (…
I do appreciate you addressing the concerns about traffic hijacking, but at the same time I really don't like having my content run through a text mangler like an LLM. I get the use case, but at the end of the day it's my content and I'm a bit prickly.
That said, I'm not necessarily planning to immediately block your crawlers, I intend to just add them to a list I maintain for personal reference. I'm mostly interested in correlating the crawling traffic that I see with various sources, I have been gathering data about crawling activity and sources that I display on an embedded map on my site. I have caddy annotate traffic with a header indicating what the crawler is, and if the fleet behaves nicely then they don't get added to the blocklist.
Re: Now AI agents need what RSS does
#68Earlier quoted context omitted.
This is going to happen, but it's too expensive for your LLM to do the scanning, and instead someone needs to build and maintain the index while allowing other people to subscribe to concepts. The problem is no one has sorted out the embedding space this all lives in.
I'd argue LLMs are getting cheaper, so it will get more feasible for LLMs to soon act on our behalf, bringing only what we're interested in. Projects like OpenClaw and Hermes already show that this can work whether the source is RSS or simply a website the agent visits. Even Google now envisions this, since they recently announced "information agents" ( https://blog.google/products-and-platforms/products/search/s...…
keywords are a start but not enough imo - consider a concept subscription such as "any of my political representatives making statements about firearm control"
Re: Now AI agents need what RSS does
#69I guess if you want your content all slurped up and served as coming from AI with no backlinks.
[flagged]
Re: Now AI agents need what RSS does
#70Earlier quoted context omitted.
I presume you’re politely asking in order to block? Which is fine, I get it. On my phone right now but can update later. I do want to ask though (and I should make this clear in a FAQ or something): the way I check RSS feeds uses adaptive scheduling, so I intentionally don’t check feeds of sites too rapidly. Then the summarization is based on the full article content but I never render that full content on the site (…
I do appreciate you addressing the concerns about traffic hijacking, but at the same time I really don't like having my content run through a text mangler like an LLM. I get the use case, but at the end of the day it's my content and I'm a bit prickly. That said, I'm not necessarily planning to immediately block your crawlers, I intend to just add them to a list I maintain for personal reference. I'm mostly intereste…
Interesting. in terms of "crawling", the way the engine I built works is by default it's just polling the rss feed of a site on an adjusting cadence like any other rss feed reader. On some sites, the engine can do a follow up scrape of the article link from the rss feed if the full content of the article isn't provided in the rss feed. So it's not real crawling, more fetching/scraping if necessary.
But I hear you.