Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scraperr – A Self Hosted Webscraper

github.com

Post reply on HN
Print view View original

41–50 of 96 posts

Re: Scraperr – A Self Hosted Webscraper

#41

Post by throwup238 »

cdolan wrote: post #40
throwup238 wrote: post #31

Earlier quoted context omitted.

llm-scraper [1] does a decent job but it's still a bit fragile. The biggest problem I have is all the React CSS-in-JS libraries that use hashes in their class names, which the LLM isn't smart enough to ignore. [1] https://github.com/mishushakov/llm-scraper

What have you had success doing with this? Curious to test it

I mostly use it to aggregate event calendars for all the concert/sport/etc venues, meetups, and clubs in my area and do some other scraping tasks. I host a little wrapper around llm-scraper on a DigitalOcean droplet that I call from Val.town scripts

I only check most places once a week so I use the LLM to do the scraping but there are a few cases where I have to scrape thousands of pages very frequently so I use the more deterministic script it generates instead.

Re: Scraperr – A Self Hosted Webscraper

#42

Post by lyjackal »

tengbretson wrote: post #35

Anyone have any experience webscraping from a Starlink IP? My assumption is you could stay under the radar due to cg nat, but it's not exactly something I want to be the first to find out about.

Mobile 4g USB sticks you can usually rotate your IP address by reconnecting. I tried on a pi, it was inconsistent. This was just with some random test mobile plan from rando carrier renting off Verizon I think

Re: Scraperr – A Self Hosted Webscraper

#43

Post by cdolan »

throwup238 wrote: post #41
cdolan wrote: post #40

Earlier quoted context omitted.

What have you had success doing with this? Curious to test it

I mostly use it to aggregate event calendars for all the concert/sport/etc venues, meetups, and clubs in my area and do some other scraping tasks. I host a little wrapper around llm-scraper on a DigitalOcean droplet that I call from Val.town scripts I only check most places once a week so I use the LLM to do the scraping but there are a few cases where I have to scrape thousands of pages very frequently so I use the…

Great thanks!

Re: Scraperr – A Self Hosted Webscraper

#44

Post by dotancohen »

anxman wrote: post #21
dotancohen wrote: post #13

Earlier quoted context omitted.

How do you work around pop-ups for newsletters and such? Look at the BBC for a good example.

Pack ad blockers into your containers. They can be loaded into Chrome and help immensely in suppressing popovers while crawling.

Thank you, I'll experiment with that. Tips and advice welcome!

Re: Scraperr – A Self Hosted Webscraper

#47

Post by beatthatflight »

VladVladikoff wrote: post #25
lucb1e wrote: post #19

Funny, I saw this HN headline just after banning another scraper's IP range You're welcome to scrape my sites but please do it ethically. Idk how to define that but some examples of things I consider not cool: - Scraping without a contact method, or at least some unique identifier (like your project's codename), in the user agent string. This is common practice, see e.g.: https://en.wikipedia.org/wiki/User-Agent_head…

What sort of pages require 20 seconds to generate? This is extremely slow by most web standards and even your users would be frustrated by this. It sounds like poorly designed database queries with unindexed joins. Google will also abandon page loads that take too long, and will demote rankings for that page (or the entire site!)

So what about flight searches where we have to query several 3rd party providers, and can take 45 seconds to get results from all of them (out of my control). I can dynamically update the page (and do) but a scraper would have to wait 20-45 seconds to get the 'cheapest' flight from my site. I can add async the queries and have the fastest pipes, but if the upstream providers take their time (they need to query their GDSs as well), there's not much you can do.

Re: Scraperr – A Self Hosted Webscraper

#48

Post by kristopolous »

windexh8er wrote: post #14
_QrE wrote: post #4

Is there a reason for using Selenium over something like Playwright? I haven't had very many positive experiences with selenium, and playwright I found is easier to use and more flexible. Also, for stuff like this: `modified_value = original_value.replace("HeadlessChrome", "Chrome")` There's quite a few ways to figure out that a browser is a bot, and I don't think replacing a few values like this does much. Not askin…

If you're a fan of Playwright check out Crawlee [0]. I've used it for a few small projects and it's been faster for me to get what I've needed done. [0] https://crawlee.dev/

It's by apify which is an interesting community

Re: Scraperr – A Self Hosted Webscraper

#49

Post by Tokumei-no-hito »

nomilk wrote: post #26
welanes wrote: post #24

Earlier quoted context omitted.

1. Clicking the box programmatically – possible but inconsistent 2. Outsourcing the task to one of the many CAPTCHA-solving services (2Captcha etc) – better 3. Using a pool of reliable IP addresses so you don't encounter checkboxes or turnstiles – best I run a web scraping startup ( https://simplescraper.io ) and this is usually the approach[0]. It has become more difficult, and I think a lot of the AI crawlers are p…

That's awesome. Thanks for sharing. First time hearing of the fetch() approach! If I understand correctly, regular browser automation might typically involve making separate GET requests for each page. Whereas the fetch() strategy involves making a GET for the first page (just as with regular browser automation), then after satisfying cloudflare, rather than going on to the next GET request, use fetch( ) to retrieve…

first time hearing about fetch too. but i don't see the advantage. is fetch reusing the connection and a manual page load not?
Post reply on HN