Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scraping 241 UK council planning portals – 2.6M decisions so far

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 76 posts

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#51

Post by mebkorea »

lifeisstillgood wrote: post #49

Some thoughts 1. Brilliant! Governments (and corps) treat public data like it’s theirs not ours. Information yearns to be free. 2. Having said that, you are likely violating T&Cs by scraping at all. 3. It is a lot easier to defend your position if you are making it free and public yourself. 4. But paying for food is nice 5. I suggest the business model here is providing architects and lawyers with strong evidence of…

Cheers, appreciate the feedback. The architect/consultant precedent angle is interesting and a couple of other commenters have already nudged me in similar directions. Tbh... you're likely right that the strongest commercial play isn't B2C £19 reports, it's giving someone fighting a contested case the national pattern across 15 similar pub conversions, the appeal outcomes, what stuck and what didn't. That's a very different product to what I have now but the data supports it. On the T&Cs/legal stuff... I'm not going to pretend I have perfect clarity on it. The position I'd defend is that the data is statutorily public, councils are required by law to publish it, I respect rate limits, and I'm aggregating not republishing in bulk. But there is this grey area between data being public to view and being usable for a commercial product, and I haven't fully nailed it down.

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#52

Post by codeulike »

mebkorea wrote: post #47
codeulike wrote: post #45

I'd be careful because even though its 'public' data, scraping it might not be legal due to TOS of the various sites. I did a search for my postcode and got given results for a different area and council miles away

Thanks for the feedback. On TOS: the same answer as I gave others... the data is statutorily public, I respect rate limits. That being said, I admit it's a grey area I haven't 100% nailed down. The postcode bug is more concerning. That shouldn't happen. Do you mind sharing which postcode or city/county? It could be that it's falling back to the wrong council because I don't have data for the right one, or it's a bug…

OK I have emailed your hello@planninglens... address with screenshots

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#53

Post by mebkorea »

safehuss wrote: post #21

This is awesome! Worked on something similar albeit a different industry. For the more challenging scrapes, would highly recommend using the Chrome Devtools MCP to be able to attach the network requests, being made by the browser to the site, as context for your agent/LLM chat - this approach really helped me to write a solid API-based scraper (also using curl_cffi) and bypassed the old tedious playwright-based appro…

Nice thinking. Hadn't thought of DevTools MCP that way. Curl_cffi I've used for TLS fingerprinting (Edinburgh was the first one) but the discovery side I've been doing manually... open DevTools, look at the request, copy as cURL, work out which params can be pruned. Automating that loop with an LLM in the middle would speed things up a lot, especially for the bespoke long tail. Will look into that this week. Thanks!

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#54

Post by mebkorea »

pbhjpbhj wrote: post #24

Have you spoken to any planners, a quick search for similar applications in other LAs might be a useful thing for them. There's a Royal Institute of Town Planners, they probably have a magazine you could advertise in (but equally that might get you blocked, idk). RICS people could probably use the data too? I guess it's useful house-buyer info; houses in the vicinity had successful loft conversions, say. On the data…

All good points. I've been so busy with the data collection and just "irl stuff" that I haven't spoken to planners directly which is an oversight on my part — they're the obvious power users. RTPI/RICS are both on my list but as I said, I've been focusing on data more than distribution. Probably the wrong order tbh. FOI is interesting, especially for the trickier portals (Liverpool's WAF, the dead-portal ones). It might be cleaner than scraping. MP/open-data angle is definitely something I hadn't seriously considered. Worth thinking about tho! Thanks.

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#55

Post by mebkorea »

codeulike wrote: post #52
mebkorea wrote: post #47

Earlier quoted context omitted.

Thanks for the feedback. On TOS: the same answer as I gave others... the data is statutorily public, I respect rate limits. That being said, I admit it's a grey area I haven't 100% nailed down. The postcode bug is more concerning. That shouldn't happen. Do you mind sharing which postcode or city/county? It could be that it's falling back to the wrong council because I don't have data for the right one, or it's a bug…

OK I have emailed your hello@planninglens... address with screenshots

Thanks. Will look into that right away!

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#56

Post by lifeisstillgood »

mebkorea wrote: post #51
lifeisstillgood wrote: post #49

Some thoughts 1. Brilliant! Governments (and corps) treat public data like it’s theirs not ours. Information yearns to be free. 2. Having said that, you are likely violating T&Cs by scraping at all. 3. It is a lot easier to defend your position if you are making it free and public yourself. 4. But paying for food is nice 5. I suggest the business model here is providing architects and lawyers with strong evidence of…

Cheers, appreciate the feedback. The architect/consultant precedent angle is interesting and a couple of other commenters have already nudged me in similar directions. Tbh... you're likely right that the strongest commercial play isn't B2C £19 reports, it's giving someone fighting a contested case the national pattern across 15 similar pub conversions, the appeal outcomes, what stuck and what didn't. That's a very di…

I agree on the “public” data issue - I spent a long time campaigning for better FOSS / data access in government and there are some great people pushing in and outside local and central gov.

But it’s a big mindset chnage (one that will benefit the whole country) but it’s slow.

I think the “push for public policy improvements” angle if genuine will get you a lot more respect and kudos when things get sticky. Good luck

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#57

Post by jayelbe »

As somebody who works in local government IT, consistent scraping of our data like this is the bane of our life. We get hit by thousands of these, many with no rate limiting, making hugely intensive requests, that cause downtime and knock-on effects for actual customers and citizens. We block IPs, add captchas, and yet it persists.

If you really want the data, just FOI it for goodness' sake.

I get the distinct impression that many of these outfits aren't really advocating for impoved transparency but are simply trying to exploit and monetise illicitly obtained government data to make a quick buck.

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#58

Post by jaggs »

I would try and go open source as fast as possible before a legal letter land on your desk. Then worry about the commercialisation. Also I have a feeling you could charge SERIOUS coin for some app for property developers based around this. But someone is almost certainly going to come at you because, you know, us Brits hate clever clogs.

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#59

Post by mebkorea »

jayelbe wrote: post #57

As somebody who works in local government IT, consistent scraping of our data like this is the bane of our life. We get hit by thousands of these, many with no rate limiting, making hugely intensive requests, that cause downtime and knock-on effects for actual customers and citizens. We block IPs, add captchas, and yet it persists. If you really want the data, just FOI it for goodness' sake. I get the distinct impres…

Fair points and yeah you must be sick of unrate-limited mass scraping. I run with 1.5-3 second delays from a single residential IP and back off when portals push back, but from your side I look the same as someone hammering you. On your point regarding FOI, what you say is fair. I should probably have led with that for the trickier councils. But the honest reason I haven't is doing 240 FOI requests at scale felt like it'd put a different kind of strain on councils, but if you're telling me the scraping is worse then I take that seriously. On "monetise illicitly obtained data"... I'm not going to pretend the £19 is altruism. But there is a public interest in this data being navigable across council boundaries, and that's not something individual councils can deliver. I must stress that I'm not sure I've got the model right yet and a lot of feedback today is pushing me toward more free, which I'm seriously considering.

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#60

Post by sublinear »

jayelbe wrote: post #57

As somebody who works in local government IT, consistent scraping of our data like this is the bane of our life. We get hit by thousands of these, many with no rate limiting, making hugely intensive requests, that cause downtime and knock-on effects for actual customers and citizens. We block IPs, add captchas, and yet it persists. If you really want the data, just FOI it for goodness' sake. I get the distinct impres…

Maybe I'm just naive, but why wouldn't a citizen do both?

I'm not implying that anything would get deliberately redacted, but it seems likely that information released through other channels would not match the web. A request might also reveal information that was not on the web.

What other choices are there?

Post reply on HN