51–60 of 96 posts
pretty cool seeing people still tweak their own scraping tools, but the cat and mouse game never ends huh - you think the web ever gets more open again or just keeps locking down?
Earlier quoted context omitted.
> It sounds like poorly designed database queries with unindexed joins Neither of those assumptions are correct. As an example, one page needs to look through 2.5 million records to find where the world record holder changed because it provides stats on who held the most records, held them for the greatest cumulative time, etc. The only thing to do would be introducing caching layers for parts of the computation, but…
Like I said then, you need indexes on those columns which you filter on in this table. Search a table of 2.5 million records for a value is still blazing fast if you use indexes correctly. I’m talking about 0.01 seconds or less. Even with tables much larger. I agree about Google being shit. However, my website makes my living, and feeds and clothes my children, so I have to play along to their rules, or suffer. Pleas…
I used to scrape back in the day when it was easy (literally just make a request and parse html). Seems cloudflare checkboxes / human verification are very commonplace nowdays. Curious how(/if) web scrapers get around those?
Does anyone know of a scraper that uses LLMs/natural language to build a deterministic, robust script that I can use to scrape the same site in the future? All of the natural language extractors I’ve seen so far need an LLM every time, but that seems unnecessary…
Earlier quoted context omitted.
1. Clicking the box programmatically – possible but inconsistent 2. Outsourcing the task to one of the many CAPTCHA-solving services (2Captcha etc) – better 3. Using a pool of reliable IP addresses so you don't encounter checkboxes or turnstiles – best I run a web scraping startup ( https://simplescraper.io ) and this is usually the approach[0]. It has become more difficult, and I think a lot of the AI crawlers are p…
That's awesome. Thanks for sharing. First time hearing of the fetch() approach! If I understand correctly, regular browser automation might typically involve making separate GET requests for each page. Whereas the fetch() strategy involves making a GET for the first page (just as with regular browser automation), then after satisfying cloudflare, rather than going on to the next GET request, use fetch( ) to retrieve…
Earlier quoted context omitted.
Like I said then, you need indexes on those columns which you filter on in this table. Search a table of 2.5 million records for a value is still blazing fast if you use indexes correctly. I’m talking about 0.01 seconds or less. Even with tables much larger. I agree about Google being shit. However, my website makes my living, and feeds and clothes my children, so I have to play along to their rules, or suffer. Pleas…
You need to drive and fine tune a Ferrari because it feeds your family. The OP just drive a nice little car, because it is fun to drive and he enjoys it. He could extract another +5% of torque by fine tuning, but he does not care, this is not where his joy is and where he wants to spend his time.
My preferred "self-hosted" webscraper is a local, single binary called xidel [1]. The feature I really like is that it can also follow links. [1] https://github.com/benibela/xidel
Wow, it's written in Pascal! That surely brings me to memory lane.
I used to scrape back in the day when it was easy (literally just make a request and parse html). Seems cloudflare checkboxes / human verification are very commonplace nowdays. Curious how(/if) web scrapers get around those?
I would prefer if we'd build a programmable web that provides value without relying on "scraping" websites for content. Most applications that do this are not well intended.