Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scraping 241 UK council planning portals – 2.6M decisions so far

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 76 posts

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#41

Post by morkee »

I hate to be a downer but...

> UK planning data is technically public.

it's public, but still copyrighted by those who submitted it

the councils also have database rights over their database, unless you've obtained explicit permission from them directly

https://en.wikipedia.org/wiki/Database_right#United_Kingdom

> I ended up writing several scrapers: a standard requests-based one, a Playwright-based one for councils that block anything that doesn't look like a real browser, and a curl_cffi one for TLS fingerprinting.

so they're explicitly trying to stop you doing this, and ... you're openly admitting to bypassing their technical measures to try and stop you?

have you heard of the Computer Misuse Act?

I doubt the 240 councils are going to be happy once they find out you've done this, especially if you're selling it on for profit

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#42

Post by nopurpose »

mebkorea wrote: post #37
vr46 wrote: post #23

Amazing! It’s so bloody hard to access this information or even to know what there is. Careful not to expose the councils too publicly before they shut you off

Cheers! Yeah, it's honestly mental how fragmented it is. Every council is its own little island. On the shutting-off worry: the data is statutorily public. Councils are legally required to publish it, and I'm respecting rate limits and not hammering anyone. So far no council has objected. Touch wood this remains the case. Tbh, I think the risk is more from the platform vendors than the councils themselves. It seems I…

There was a story how similar initiative for a courts decisions scraping was shut off.

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#43

Post by simonjuk »

I work with public data, and I'd love to get access to this data, but I suspect that although you have scraped the data from public websites, there are licensing and copyright implications for actually using it.

See also the open addresses project by Data Adaptive [1] which is using Freedom of Information requests to publish public council tax address data. The problem they have run into there is that their address datasets are derived from proprietary Ordnance Survey data.

It looks like data.gov.uk is in the process of standardising the planning application process, and publishing them under OGL [2].

[1]: https://www.owenboswarva.com/blog/post-addr44.htm [2]: https://www.planning.data.gov.uk/dataset/planning-applicatio...

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#44

Post by mebkorea »

morkee wrote: post #41

I hate to be a downer but... > UK planning data is technically public. it's public, but still copyrighted by those who submitted it the councils also have database rights over their database, unless you've obtained explicit permission from them directly https://en.wikipedia.org/wiki/Database_right#United_Kingdom > I ended up writing several scrapers: a standard requests-based one, a Playwright-based one for councils…

Fair points and I appreciate the feedback. Database right is real but the threshold is "substantial part". I'm literally only showing aggregates and letting people search by postcode. I'm not completely republishing council databases. Think that's defensible, but not gonna pretend that it's 100% black and white. On CMA, I'd push back. That's about unauthorised access. These portals are public-facing and the data's published deliberately for people to view. Rotating user-agents isn't bypassing security in any meaningful way... I'm not breaking auth or guessing passwords. I back off when portals signal they're unhappy (Liverpool's WAF actively rate-limited me which is why that data's stale). No council has reached out so far. Could change ofc. Solo founder with no legal team though, so happy to be told I've got it wrong.

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#46

Post by mebkorea »

simonjuk wrote: post #43

I work with public data, and I'd love to get access to this data, but I suspect that although you have scraped the data from public websites, there are licensing and copyright implications for actually using it. See also the open addresses project by Data Adaptive [1] which is using Freedom of Information requests to publish public council tax address data. The problem they have run into there is that their address d…

Thanks and yeah, some of my boundary data (for the choropleth) comes from ONS open boundary files which I think are OGL but I'd need to check the chain of derivation. On the data.gov.uk standardisation, I've seen it but last I looked it was policy and boundaries, not actual decisions. Has that changed? If they're publishing decisions under OGL I'd gladly ditch the scraping for a proper feed. On licensing more generally... I haven't fully nailed it down. Showing aggregates and pointing back to source, but yeah there's a gap between data is public and do whatever you want with it commercially.

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#47

Post by mebkorea »

codeulike wrote: post #45

I'd be careful because even though its 'public' data, scraping it might not be legal due to TOS of the various sites. I did a search for my postcode and got given results for a different area and council miles away

Thanks for the feedback. On TOS: the same answer as I gave others... the data is statutorily public, I respect rate limits. That being said, I admit it's a grey area I haven't 100% nailed down. The postcode bug is more concerning. That shouldn't happen. Do you mind sharing which postcode or city/county? It could be that it's falling back to the wrong council because I don't have data for the right one, or it's a bug in my mapping. Either way, it needs fixing asap! Cheers for flagging.

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#48

Post by mebkorea »

ferngodfather wrote: post #16

Your terms: > You may not use automated tools to scrape, copy, or bulk-download data from our service. Pot kettle, huh.

Updated and pushed live: planninglens.co.uk/terms. Acceptable Use clause now permits programmatic access that respects rate limits, while still protecting our derived analysis and reports. Thanks for the kick.

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#49

Post by lifeisstillgood »

Some thoughts

1. Brilliant! Governments (and corps) treat public data like it’s theirs not ours. Information yearns to be free.

2. Having said that, you are likely violating T&Cs by scraping at all.

3. It is a lot easier to defend your position if you are making it free and public yourself.

4. But paying for food is nice

5. I suggest the business model here is providing architects and lawyers with strong evidence of prior planning decisions nationally

Most people applying for (difficult) planning have experience locally. But the planning system is a mess because it is not coherent nationally or regionally. The win here is not providing a copy of your data (that has legal issues) but providing pointers to decisions that support the case of the person paying you.

So I want to turn an old pub into tasteful housing and a cafe for the local village. The local planning team don’t like it, I could spend money bribing them and the councillors (see how much I understand British democracy) or I could get from you the fifteen pub to housing conversion decisions from around the country and use that to help my bribed councillors defend their u-turn

Everyone wins :-)

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#50

Post by mebkorea »

CJefferson wrote: post #2

So, this sounds exciting to me, but the postcode checker really feels like a spam as a user. All it tells me is 'Mixed results'. I could make a website that prints 'mixed results', I bet most results are 'mixed'! I understand wanting to get money, but honestly, there is no way I would give money to this website in it's current state, you are giving me far too little info before asking me to hand over a credit card. T…

[dead]
Post reply on HN