41–50 of 63 posts
My instinct was also to use LLMs for this, but it was way to slow and still expensive if you want to scrape millions of pages.
My instinct was also to use LLMs for this, but it was way to slow and still expensive if you want to scrape millions of pages.
Put things to perspective - Gemini 2.5 flash is 0.3/1M tokens - assuming each page is 700 tokens and output is not much you are looking at $210 for 1M pages
Edit: There's also the added complexity of running a browser against 1M pages, or more.
I need to extract article content, determine it's sentiment towards a keyword and output a simple json with article name, url, sentiment and some text around the found keyword.
Currently I'm having problems with the json output, it's not reliable enough and produces a lot of false json.
My platform has 24M pages on 8 domains and these NASTY crawlers insist on visiting every single one of them. For every 1 real visitor there are at least 300 requests from residential proxies. And that's after I blocked complete countries like Russia, China, Taiwan and Singapore. Even Cloudflares bot filter only blocks some of them. I'm using honeypot URLs right now to block all crawlers that ignore rel="nofollow", bu…
My platform has 24M pages on 8 domains and these NASTY crawlers insist on visiting every single one of them. For every 1 real visitor there are at least 300 requests from residential proxies. And that's after I blocked complete countries like Russia, China, Taiwan and Singapore. Even Cloudflares bot filter only blocks some of them. I'm using honeypot URLs right now to block all crawlers that ignore rel="nofollow", bu…
What crawlers are using residential proxies?
I'd put my money on Chinese AI model makers, but I don't trust any company that is in desperate need of fresh data.
> LLMs return malformed JSON more often than you'd expect, especially with nested arrays and complex schemas. One bad bracket and your pipeline crashes. This might be one reason why Claude Code uses XML for tool calling: repeating the tag name in the closing bracket helps it keep track of where it is during inference, so it is less error prone.