Show HN: Robust LLM extractor for websites in TypeScript • Hacker News Board

Re: Show HN: Robust LLM extractor for websites in TypeScript

#61

Post by andrew_zhong » Sat, Mar 28, 2026, 4:15 PM UTC

Ryand1234 wrote: post #60

Hey, 1 question, does it extract interactive data too? I mean data that is visible after interaction, like the collapse bar and others?

You can use a browser automation library for rendering JS + interaction (like click collapse button) and then use this library to extract the HTML after interaction. Here is an example of using a AI browser automation library with prompt to interact (but you can also use playwright if you know the exact element to interact)

https://github.com/lightfeed/extractor?tab=readme-ov-file#us...

Re: Show HN: Robust LLM extractor for websites in TypeScript

#62

Post by rsafaya » Sun, Mar 29, 2026, 12:22 PM UTC

Maybe it's time scrapers actually paid publishers via something like HTTP 402 for their data instead of an arms race with Cloudflare on one side and residential proxies on the other.

Re: Show HN: Robust LLM extractor for websites in TypeScript

#63

Post by vetler » Mon, Mar 30, 2026, 10:29 AM UTC

andrew_zhong wrote: post #52

vetler wrote: post #43

Earlier quoted context omitted.

You will absolutely struggle to get all the info you need into 700 tokens per page. Edit: There's also the added complexity of running a browser against 1M pages, or more.

I agree that When pages have similar structure, for one time extraction as it is (not reasoning from context), scraping with selectors is the way to go. This library also supports HTML as input so running a browser is not required.

Came back here to say I was wrong! I have been experimenting, and it is doable. I have been experimenting with setting up a scraping pipeline with LLM enrichment since I wrote the comment above, and have very positive results so far. :)