Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scraperr – A Self Hosted Webscraper

github.com

Post reply on HN
Print view View original

71–80 of 96 posts

Re: Scraperr – A Self Hosted Webscraper

#71

Post by throawayonthe »

Loic wrote: post #63
whazor wrote: post #61

Earlier quoted context omitted.

I find it unethical for a website robots.txt to allow-list particular search engines and ban all others. Essentially you are colluding with established search providers.

Not necessarily, I have a website with 95% (maybe even more) of the traffic generated by crawlers. If some of them are behaving badly, it is fair to exclude them with my robots.txt. But of course, the ones behaving badly tend to not respect the robots.txt, so you end up banning the IP or IP block. And here, I am a nice guy, the crawler must really be a piece of crap for me to start to block.

the parent comment is talking about allow-listing (aka 'whitelisting') just a few crawlers from like, google

Re: Scraperr – A Self Hosted Webscraper

#72

Post by monkeydust »

Practical use-case.

I am looking for a way to throw an address at a planning authority (UK) and download the associated documents for that property. Could this or another tool help?

e.g.

https://publicaccess.barnet.gov.uk/online-applications/appli...

As pure random example.

A property can have multiple planning applications and under each many documents.

What I have found useful (saved me time and potential lost £££) is to take the documents, combine to single pdf and provide to Gemini 2.5 Pro and then ask it to validate against agent specification for a property.

Over the weekend found a place that was advertising a feature of the house that was explicitly prohibited through planning decision notice.

Called the Agent up on it who claimed no knowledge but said this would have come up through solicitor checks, which it would have done, much later down the process with more or my money spent and considerable time lost.

Of course all this possible without LLMs but just makes it easier/cheaper to check at scale.

Re: Scraperr – A Self Hosted Webscraper

#73

Post by cess11 »

monkeydust wrote: post #72

Practical use-case. I am looking for a way to throw an address at a planning authority (UK) and download the associated documents for that property. Could this or another tool help? e.g. https://publicaccess.barnet.gov.uk/online-applications/appli... As pure random example. A property can have multiple planning applications and under each many documents. What I have found useful (saved me time and potential lost £££)…

Could just cut out the href-value with grep and sed or a bit of scripting, '.pdf' seems to only occur on those links.

I'd keep it simple like that until I need to do periodic comparisons, i.e. actually need scrapers and is prepared to build what's needed to automatically watch and process directories where the scrapers put the files.

Re: Scraperr – A Self Hosted Webscraper

#74

Post by cess11 »

nomilk wrote: post #18

I used to scrape back in the day when it was easy (literally just make a request and parse html). Seems cloudflare checkboxes / human verification are very commonplace nowdays. Curious how(/if) web scrapers get around those?

Low effort baseline would be https://seleniumbase.io/, to drive a preconfigured web browser that looks relatively human to the network service. Typically it just clicks through the one-click captcha:s.

If that's not good enough you'll likely have to fiddle with your own web driver and possibly a computer vision rig to manage to click through 'find the motorcycle' kind of challenges. Paying a click farm to do it for you is probably cheaper in the short run.

An important hurdle is getting reputable IPv4 addresses to do it from, if you're going to do it a lot. Having or renting a botnet could help, but might be too illegal for your use case.

Re: Scraperr – A Self Hosted Webscraper

#75

Post by cess11 »

3abiton wrote: post #11

> extract data from websites with precision using XPath selectors. I've used XPath for crawling with selenium, and it used to be my favorite way, but turned out quite unreliable if you don't combine it with other selectors as certain website are really badly designed and have no good patterns. So what's the added value over pure selenium?

Check whether the site is actually server side rendered, because if it's a browser client that talks JSON to the backend, you could do the same.

Re: Scraperr – A Self Hosted Webscraper

#76

Post by p3rls »

ToucanLoucan wrote: post #69
whazor wrote: post #61

Earlier quoted context omitted.

I find it unethical for a website robots.txt to allow-list particular search engines and ban all others. Essentially you are colluding with established search providers.

This rather bluntly runs up against the fact that permitting crawling is an expense the web operator is taking on, ergo, receiving that content is by definition a privilege not a right.

I think my team has spent about a quarter of this year's dev time on defending against scrapers. Their arrogance is the stuff of ancient greek dramas.

Re: Scraperr – A Self Hosted Webscraper

#77

Post by monooso »

VladVladikoff wrote: post #34
lucb1e wrote: post #27

Earlier quoted context omitted.

> It sounds like poorly designed database queries with unindexed joins Neither of those assumptions are correct. As an example, one page needs to look through 2.5 million records to find where the world record holder changed because it provides stats on who held the most records, held them for the greatest cumulative time, etc. The only thing to do would be introducing caching layers for parts of the computation, but…

Like I said then, you need indexes on those columns which you filter on in this table. Search a table of 2.5 million records for a value is still blazing fast if you use indexes correctly. I’m talking about 0.01 seconds or less. Even with tables much larger. I agree about Google being shit. However, my website makes my living, and feeds and clothes my children, so I have to play along to their rules, or suffer. Pleas…

You have very strong opinions about a site you effectively know nothing about.

Instead of immediately concluding that the person actually building the system is an incompetent fool who doesn't know any better, maybe work on the assumption that they know what they're doing, and have already considered the various trade-offs.

If nothing else, that would be considerably less obnoxious.

Re: Scraperr – A Self Hosted Webscraper

#78

Post by yoble »

andrethegiant wrote: post #37

Shameless plug: prefix any URL with https://pure.md/ to get the pure markdown of that page. Useful for direct piping into an LLM. Has bot detection avoidance, proxy rotation, and headless JS rendering built in.

Love the easter egg when going to https://pure.md/https://pure.md

Re: Scraperr – A Self Hosted Webscraper

#79

Post by Fokamul »

lucb1e wrote: post #19

Funny, I saw this HN headline just after banning another scraper's IP range You're welcome to scrape my sites but please do it ethically. Idk how to define that but some examples of things I consider not cool: - Scraping without a contact method, or at least some unique identifier (like your project's codename), in the user agent string. This is common practice, see e.g.: https://en.wikipedia.org/wiki/User-Agent_head…

Who cares, IP ranges are cheap. You're just banning datacenters.

Re: Scraperr – A Self Hosted Webscraper

#80

Post by datatrashfire »

lucb1e wrote: post #19

Funny, I saw this HN headline just after banning another scraper's IP range You're welcome to scrape my sites but please do it ethically. Idk how to define that but some examples of things I consider not cool: - Scraping without a contact method, or at least some unique identifier (like your project's codename), in the user agent string. This is common practice, see e.g.: https://en.wikipedia.org/wiki/User-Agent_head…

> - Scraping without a contact method, or at least some unique identifier (like your project's codename), in the user agent string.

This is a very effective way to make sure you won't get any scraping done!

Post reply on HN