21–30 of 63 posts
Robots.txt anyone?
Good point. The anti-bot patches here (via Patchright) are about preventing the browser from being detected as automated — things like CDP leak fixes so Cloudflare doesn't block you mid-session. It's not about bypassing access restrictions. Our main use case is retail price monitoring — comparing publicly listed product prices across e-commerce sites, which is pretty standard in the industry. But fair point, we shoul…
> Avoid detection with built-in anti-bot patches and proxy configuration for reliable web scraping. And it doesn't care about robots.txt.
Good point. The anti-bot patches here (via Patchright) are about preventing the browser from being detected as automated — things like CDP leak fixes so Cloudflare doesn't block you mid-session. It's not about bypassing access restrictions. Our main use case is retail price monitoring — comparing publicly listed product prices across e-commerce sites, which is pretty standard in the industry. But fair point, we shoul…
[0]: https://github.com/lightfeed/extractor/blob/d11060269e65459e...
Earlier quoted context omitted.
Good point. The anti-bot patches here (via Patchright) are about preventing the browser from being detected as automated — things like CDP leak fixes so Cloudflare doesn't block you mid-session. It's not about bypassing access restrictions. Our main use case is retail price monitoring — comparing publicly listed product prices across e-commerce sites, which is pretty standard in the industry. But fair point, we shoul…
Regardless. You should still respect robots.txt..
I will add a PR to enforce robots.txt before the actual scraping.
Earlier quoted context omitted.
Good point. The anti-bot patches here (via Patchright) are about preventing the browser from being detected as automated — things like CDP leak fixes so Cloudflare doesn't block you mid-session. It's not about bypassing access restrictions. Our main use case is retail price monitoring — comparing publicly listed product prices across e-commerce sites, which is pretty standard in the industry. But fair point, we shoul…
> It's not about bypassing access restrictions. Yes. It is. You've just made an arbitrary choice not to define it as such.
Earlier quoted context omitted.
Regardless. You should still respect robots.txt..
We do respect robots.txt production - also scraping browser providers like BrightData enforces that. I will add a PR to enforce robots.txt before the actual scraping.
This feels like slop to me. It may or may not be, but if you want people to actually use this product I’d suggest improving your documentation and replies here to not look like raw Claude output. I also doubt the premise that about malformed JSON. I have never encountered anything like what you are describing with structured outputs.
price: z.number().optional() -> price: “n/a”
url: z.string().url().nullable() -> url: “not found”
It can also be one invalid object (e.g. missing required field, truncated input) in an array causing the entire output to fail.
The unique contribution here is we can recover invalid nullable or optional field, and also remove invalid nested objects in an array.
This looks pretty interesting! I haven't used it yet, but looked through the code a bit, it looks like it uses turndown to convert the html to markdown first, then it passes that to the LLM so assuming that's a huge reduction in tokens by preprocessing. Do you have any data on how often this can cause issues? ie tables or other information being lost? Then langchain and structured schemas for the output along w/ a sp…
https://github.com/lightfeed/extractor/blob/main/src/convert...
I have used gemma 3 and had good results.
Once Gemini 3 flash drops the preview suffix, will update the examples. Thank you for the pointer.
> LLMs return malformed JSON more often than you'd expect, especially with nested arrays and complex schemas. One bad bracket and your pipeline crashes. This might be one reason why Claude Code uses XML for tool calling: repeating the tag name in the closing bracket helps it keep track of where it is during inference, so it is less error prone.
Also, a model can always use a proxy to turn your tool calls into XML
And feed you back json right away and you wouldn't even know if any transformation did take place.