Viewing profile — andrew_zhong

Systematic testing of browser agents today is not easy: testing on real websites is flaky, rate-limited and potentially expensive (e.g. using proxies or bypassing Captcha), while s…

HTML-to-Markdown converters produce clean, readable content for both humans and LLMs — but the DOM structure is lost along the way. You can always feed Markdown to an LLM to extrac…

You can use a browser automation library for rendering JS + interaction (like click collapse button) and then use this library to extract the HTML after interaction. Here is an exa…

[Update]] I will replace the stealth browser with plain playwright and remove anti-bot as a feature.

I hear you loud and clear - will replace the stealth browser with plain playwright and remove anti-bot as a feature.

[Update] I will replace the stealth browser with plain playwright and remove anti-bot as a feature.

I agree that When pages have similar structure, for one time extraction as it is (not reasoning from context), scraping with selectors is the way to go. This library also supports …

What kind of LLMs are you using? In structured output mode? In this library we recover nullable and optional fields, invalid elements in nested array, bad urls, repair incomplete J…

Put things to perspective - Gemini 2.5 flash is 0.3/1M tokens - assuming each page is 700 tokens and output is not much you are looking at $210 for 1M pages

Agreed - in this project I did a one path sanitation to recover invalid optional / nullable fields or discard invalid objects in nested array. I know multi path LLM approaches exis…

We do see fewer invalid JSONs on latest bigger LLMs but still can happen on smaller and cheaper models. There is also case when input is truncated or a required field not found, wh…

HTML -> markdown -> LLM is standard practice. We strip elements like aside, embed, head , iframe etc. the criteria is conservatively set to avoid removing too many elements (especi…

In context of e-commerce web extraction, invalid JSON can occur especially in edge cases, for example: price: z.number().optional() -> price: “n/a” url: z.string().url().nullable()…

I will add a PR to enforce robots.txt before the actual scraping.

We do respect robots.txt production - also scraping browser providers like BrightData enforces that. I will add a PR to enforce robots.txt before the actual scraping.

The anti-bot patches here (via Patchright) are about preventing the browser from being detected as automated — fixing CDP leaks, removing automation flags, etc. For sites behind Cl…

Yeah that's a good observation. XML's closing tags give the model structural anchors during generation — it knows where it is in the nesting. JSON doesn't have that, so the deeper …

Good point. The anti-bot patches here (via Patchright) are about preventing the browser from being detected as automated — things like CDP leak fixes so Cloudflare doesn't block yo…

Good point. The anti-bot patches here (via Patchright) are about preventing the browser from being detected as automated — things like CDP leak fixes so Cloudflare doesn't block yo…

We've been building data pipelines that scrape websites and extract structured data for a while now. If you've done this, you know the drill: you write CSS selectors, the site chan…

While working with LLMs for structured web data extraction, we saw issues with invalid JSON and broken links in the output. This led me to build a library focused on robust extract…

Thank you for the feedback. Our intention is to let users try for 10 days without commitment (no credit card required), so that after 10 days they can choose to subscribe or not. I…