Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Robust LLM extractor for websites in TypeScript

github.com

Post reply on HN
Print view View original

51–60 of 63 posts

Re: Show HN: Robust LLM extractor for websites in TypeScript

#51

Post by andrew_zhong »

l3x4ur1n wrote: post #44

Would this work for my use case? I need to extract article content, determine it's sentiment towards a keyword and output a simple json with article name, url, sentiment and some text around the found keyword. Currently I'm having problems with the json output, it's not reliable enough and produces a lot of false json.

What kind of LLMs are you using? In structured output mode?

In this library we recover nullable and optional fields, invalid elements in nested array, bad urls, repair incomplete JSONs. If these issues are what you see, yes it should work for your case.

Re: Show HN: Robust LLM extractor for websites in TypeScript

#52

Post by andrew_zhong »

vetler wrote: post #43
andrew_zhong wrote: post #41

Earlier quoted context omitted.

Put things to perspective - Gemini 2.5 flash is 0.3/1M tokens - assuming each page is 700 tokens and output is not much you are looking at $210 for 1M pages

You will absolutely struggle to get all the info you need into 700 tokens per page. Edit: There's also the added complexity of running a browser against 1M pages, or more.

I agree that When pages have similar structure, for one time extraction as it is (not reasoning from context), scraping with selectors is the way to go.

This library also supports HTML as input so running a browser is not required.

Re: Show HN: Robust LLM extractor for websites in TypeScript

#53

Post by sheept »

olafura wrote: post #50
sheept wrote: post #7

> LLMs return malformed JSON more often than you'd expect, especially with nested arrays and complex schemas. One bad bracket and your pipeline crashes. This might be one reason why Claude Code uses XML for tool calling: repeating the tag name in the closing bracket helps it keep track of where it is during inference, so it is less error prone.

Unless I'm totally misunderstanding something it's not xml but special tokens for the tokenizer someone smarter than me might know https://medium.com/@nisarg.nargund/why-special-tokens-matter...

Not in Claude Code, where asking it to print the XML used for tool calling makes it accidentally trigger the tool call

Re: Show HN: Robust LLM extractor for websites in TypeScript

#55

Post by andrew_zhong »

plastic041 wrote: post #25
andrew_zhong wrote: post #23

Earlier quoted context omitted.

We do respect robots.txt production - also scraping browser providers like BrightData enforces that. I will add a PR to enforce robots.txt before the actual scraping.

How can people believe that you are respecting bot detection in production when your software's README says it can "Avoid detection with built-in anti-bot patches"?

I hear you loud and clear - will replace the stealth browser with plain playwright and remove anti-bot as a feature.

Re: Show HN: Robust LLM extractor for websites in TypeScript

#56

Post by andrew_zhong »

andrew_zhong wrote: post #11
zx8080 wrote: post #6

Robots.txt anyone?

Good point. The anti-bot patches here (via Patchright) are about preventing the browser from being detected as automated — things like CDP leak fixes so Cloudflare doesn't block you mid-session. It's not about bypassing access restrictions. Our main use case is retail price monitoring — comparing publicly listed product prices across e-commerce sites, which is pretty standard in the industry. But fair point, we shoul…

[Update]] I will replace the stealth browser with plain playwright and remove anti-bot as a feature.

Re: Show HN: Robust LLM extractor for websites in TypeScript

#57

Post by messe »

andrew_zhong wrote: post #24
messe wrote: post #17

Earlier quoted context omitted.

> It's not about bypassing access restrictions. Yes. It is. You've just made an arbitrary choice not to define it as such.

I will add a PR to enforce robots.txt before the actual scraping.

Or just follow web standards and define and publish your User-Agent header, so that people can block that as needed.

You're creating the wrong kind of value. I really hope your company fails, as its success implies a failure of the web in general.

I wish you the best success outside of your current endeavour.

Re: Show HN: Robust LLM extractor for websites in TypeScript

#58

Post by qingcharles »

andrew_zhong wrote: post #28
Flux159 wrote: post #4

This looks pretty interesting! I haven't used it yet, but looked through the code a bit, it looks like it uses turndown to convert the html to markdown first, then it passes that to the LLM so assuming that's a huge reduction in tokens by preprocessing. Do you have any data on how often this can cause issues? ie tables or other information being lost? Then langchain and structured schemas for the output along w/ a sp…

HTML -> markdown -> LLM is standard practice. We strip elements like aside, embed, head , iframe etc. the criteria is conservatively set to avoid removing too many elements (especially in extractMain mode) https://github.com/lightfeed/extractor/blob/main/src/convert... I have used gemma 3 and had good results. Once Gemini 3 flash drops the preview suffix, will update the examples. Thank you for the pointer.

I use for infoboxes on my wiki. Am I using aside wrong or are you stripping too heavily?

Re: Show HN: Robust LLM extractor for websites in TypeScript

#59

Post by dmos62 »

spiderfarmer wrote: post #34

My platform has 24M pages on 8 domains and these NASTY crawlers insist on visiting every single one of them. For every 1 real visitor there are at least 300 requests from residential proxies. And that's after I blocked complete countries like Russia, China, Taiwan and Singapore. Even Cloudflares bot filter only blocks some of them. I'm using honeypot URLs right now to block all crawlers that ignore rel="nofollow", bu…

Have you considered recaptcha v2 and similar? Proof of work might slow them down. Sounds pretty bad. Would be great if Cloudflare, Datadome, etc. were doing this for you and thus banning these devices for everyone.
Post reply on HN