Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Got Tired of LLMs refusing to visit URLs Built open source analyzer CLI

github.com

Post reply on HN
Print view View original

1–4 of 4 posts

Show HN: Got Tired of LLMs refusing to visit URLs Built open source analyzer CLI

#1

Post by thetall0ne »

Built this because I got tired of ChatGPT/Claude refusing to visit websites when doing research.

Crawls sitemaps, parses metadata files (robots.txt, humans.txt, llms.txt), detects tech stack with Wappalyzer, then generates summaries using either AWS Bedrock or a local Llama model via Ollama.

Batch mode processes CSVs with checkpoint/resume - hit Ctrl+C anytime and pick up where you left off.

The local LLM option means zero API costs and your data never leaves your machine. Llama 3.2 3B works surprisingly well for this task.

Show HN: Got Tired of LLMs refusing to visit URLs Built open source analyzer CLI
github.com

Re: Show HN: Got Tired of LLMs refusing to visit URLs Built open source analyzer CLI

#2

Post by mojomark »

Nice - right? How annoying is that. If a public human can read content, why can't an LLM? ChatGPT/Claude also (at least the for me, also don't consistently fully review the content I upload for review. Sometimes it's full, but most of the time (especially if it's a larger document, say 100pg pdf or 15 python scripts), I have to continually push them to go through everything.

Really annoying - thank you for this! Now, am I too lazy to apply it, that's the question.

Re: Show HN: Got Tired of LLMs refusing to visit URLs Built open source analyzer CLI

#3

Post by thetall0ne »

mojomark wrote: post #2

Nice - right? How annoying is that. If a public human can read content, why can't an LLM? ChatGPT/Claude also (at least the for me, also don't consistently fully review the content I upload for review. Sometimes it's full, but most of the time (especially if it's a larger document, say 100pg pdf or 15 python scripts), I have to continually push them to go through everything. Really annoying - thank you for this! Now,…

lol! no problem. Its such an annoying problem. One time and LLM said "I can't directly access URLs" and I replied with "yes you can" and then it did it! WTF!?
Post reply on HN