Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Robust LLM extractor for websites in TypeScript

github.com

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Show HN: Robust LLM extractor for websites in TypeScript

#31

Post by bilekas »

andrew_zhong wrote: post #11
zx8080 wrote: post #6

Robots.txt anyone?

Good point. The anti-bot patches here (via Patchright) are about preventing the browser from being detected as automated — things like CDP leak fixes so Cloudflare doesn't block you mid-session. It's not about bypassing access restrictions. Our main use case is retail price monitoring — comparing publicly listed product prices across e-commerce sites, which is pretty standard in the industry. But fair point, we shoul…

> comparing publicly listed product prices across e-commerce sites

Those prices and information is for the public viewers, the reason why some people have ROBOTS.txt for example is to reduce the traffic load that slop crawlers generate. The bandwidth is not free so why would you assume to ignore their ROBOTS.txt when you're not footing the bill ?

Re: Show HN: Robust LLM extractor for websites in TypeScript

#32

Post by spiderfarmer »

andrew_zhong wrote: post #14
dmos62 wrote: post #8

What's your experience with not getting blocked by anti-bot systems? I see you've custom patches for that.

The anti-bot patches here (via Patchright) are about preventing the browser from being detected as automated — fixing CDP leaks, removing automation flags, etc. For sites behind Cloudflare or Datadome, that alone usually isn't enough — you'll need residential proxies and proper browser fingerprints on top. The library supports connecting to remote scraping browsers via WebSocket and proxy configuration for those case…

As someone who is getting HAMMERED TO NO BELIEVE by residential proxies, I just want to express my hatred to all of you.

Re: Show HN: Robust LLM extractor for websites in TypeScript

#33

Post by [deleted] »

bilekas wrote: post #31
andrew_zhong wrote: post #11

Earlier quoted context omitted.

Good point. The anti-bot patches here (via Patchright) are about preventing the browser from being detected as automated — things like CDP leak fixes so Cloudflare doesn't block you mid-session. It's not about bypassing access restrictions. Our main use case is retail price monitoring — comparing publicly listed product prices across e-commerce sites, which is pretty standard in the industry. But fair point, we shoul…

> comparing publicly listed product prices across e-commerce sites Those prices and information is for the public viewers, the reason why some people have ROBOTS.txt for example is to reduce the traffic load that slop crawlers generate. The bandwidth is not free so why would you assume to ignore their ROBOTS.txt when you're not footing the bill ?

[deleted]

Re: Show HN: Robust LLM extractor for websites in TypeScript

#34

Post by spiderfarmer »

My platform has 24M pages on 8 domains and these NASTY crawlers insist on visiting every single one of them. For every 1 real visitor there are at least 300 requests from residential proxies. And that's after I blocked complete countries like Russia, China, Taiwan and Singapore.

Even Cloudflares bot filter only blocks some of them.

I'm using honeypot URLs right now to block all crawlers that ignore rel="nofollow", but they appear to have many millions of devices. I wouldn't be surprised if there are a gazillion residential routers, webcams and phones that are hacked to function as a simple doorways.

Things are really getting out of hand.

Re: Show HN: Robust LLM extractor for websites in TypeScript

#35

Post by andrew_zhong »

faangguyindia wrote: post #29
sheept wrote: post #7

> LLMs return malformed JSON more often than you'd expect, especially with nested arrays and complex schemas. One bad bracket and your pipeline crashes. This might be one reason why Claude Code uses XML for tool calling: repeating the tag name in the closing bracket helps it keep track of where it is during inference, so it is less error prone.

Hardly matters, this isn't a problem that you'd have these days with modern LLMs. Also, a model can always use a proxy to turn your tool calls into XML And feed you back json right away and you wouldn't even know if any transformation did take place.

We do see fewer invalid JSONs on latest bigger LLMs but still can happen on smaller and cheaper models. There is also case when input is truncated or a required field not found, which are inherently difficult.

On XML vs JSON, I think the goal here is to generate typed output where JSON with zod shines - for example the result can type check and be inserted to database typed columns later

Re: Show HN: Robust LLM extractor for websites in TypeScript

#37

Post by faangguyindia »

andrew_zhong wrote: post #35
faangguyindia wrote: post #29

Earlier quoted context omitted.

Hardly matters, this isn't a problem that you'd have these days with modern LLMs. Also, a model can always use a proxy to turn your tool calls into XML And feed you back json right away and you wouldn't even know if any transformation did take place.

We do see fewer invalid JSONs on latest bigger LLMs but still can happen on smaller and cheaper models. There is also case when input is truncated or a required field not found, which are inherently difficult. On XML vs JSON, I think the goal here is to generate typed output where JSON with zod shines - for example the result can type check and be inserted to database typed columns later

Thing is even with XML LLM will fail every now and then.

I've built an agent in both tool calling and by parsing XML

You always need a self correcting loop built in, if you are editing a file with LLM you need provide hints so LLM gets it right the second time or 3rd or n time.

Just by switching to XML you'll not get that.

I used to use XML now i only use it for examples in in system prompt for model to learn. That's all

Re: Show HN: Robust LLM extractor for websites in TypeScript

#38

Post by dmos62 »

spiderfarmer wrote: post #32
andrew_zhong wrote: post #14

Earlier quoted context omitted.

The anti-bot patches here (via Patchright) are about preventing the browser from being detected as automated — fixing CDP leaks, removing automation flags, etc. For sites behind Cloudflare or Datadome, that alone usually isn't enough — you'll need residential proxies and proper browser fingerprints on top. The library supports connecting to remote scraping browsers via WebSocket and proxy configuration for those case…

As someone who is getting HAMMERED TO NO BELIEVE by residential proxies, I just want to express my hatred to all of you.

Curious. Care to share more? What approaches have you tried?

Re: Show HN: Robust LLM extractor for websites in TypeScript

#39

Post by spiderfarmer »

dmos62 wrote: post #38
spiderfarmer wrote: post #32

Earlier quoted context omitted.

As someone who is getting HAMMERED TO NO BELIEVE by residential proxies, I just want to express my hatred to all of you.

Curious. Care to share more? What approaches have you tried?

https://news.ycombinator.com/edit?id=47528370

Re: Show HN: Robust LLM extractor for websites in TypeScript

#40

Post by andrew_zhong »

faangguyindia wrote: post #37
andrew_zhong wrote: post #35

Earlier quoted context omitted.

We do see fewer invalid JSONs on latest bigger LLMs but still can happen on smaller and cheaper models. There is also case when input is truncated or a required field not found, which are inherently difficult. On XML vs JSON, I think the goal here is to generate typed output where JSON with zod shines - for example the result can type check and be inserted to database typed columns later

Thing is even with XML LLM will fail every now and then. I've built an agent in both tool calling and by parsing XML You always need a self correcting loop built in, if you are editing a file with LLM you need provide hints so LLM gets it right the second time or 3rd or n time. Just by switching to XML you'll not get that. I used to use XML now i only use it for examples in in system prompt for model to learn. That's…

Agreed - in this project I did a one path sanitation to recover invalid optional / nullable fields or discard invalid objects in nested array.

I know multi path LLM approaches exist: e.g. generating JSON patches

https://github.com/hinthornw/trustcall

Post reply on HN