Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scraping 241 UK council planning portals – 2.6M decisions so far

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 76 posts

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#61

Post by mebkorea »

jaggs wrote: post #58

I would try and go open source as fast as possible before a legal letter land on your desk. Then worry about the commercialisation. Also I have a feeling you could charge SERIOUS coin for some app for property developers based around this. But someone is almost certainly going to come at you because, you know, us Brits hate clever clogs.

Yeah, unfortunately you are not wrong about the national tradition of wanting to pull down clever-clogs...

The open source angle is something I'm increasingly considering, especially after a local government IT person made a fair point on this thread about the strain it causes. It won't fix the scraping load directly, but might frame the project as public-interest rather than attempting to make a bit of extra money. Tbh, the real value is probably in serving property developers and consultants, not emailing £19 PDFs to homeowners. Got a lot to think about.

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#62

Post by mebkorea »

lifeisstillgood wrote: post #56
mebkorea wrote: post #51

Earlier quoted context omitted.

Cheers, appreciate the feedback. The architect/consultant precedent angle is interesting and a couple of other commenters have already nudged me in similar directions. Tbh... you're likely right that the strongest commercial play isn't B2C £19 reports, it's giving someone fighting a contested case the national pattern across 15 similar pub conversions, the appeal outcomes, what stuck and what didn't. That's a very di…

I agree on the “public” data issue - I spent a long time campaigning for better FOSS / data access in government and there are some great people pushing in and outside local and central gov. But it’s a big mindset chnage (one that will benefit the whole country) but it’s slow. I think the “push for public policy improvements” angle if genuine will get you a lot more respect and kudos when things get sticky. Good luck

Cheers, and things get sticky isn't lost on me, especially after some of the feedback. Seems all the feedback is converging on the same direction namely open-source the data layer, lead with public-interest, let any commercial product sit on top rather than be the thing. Haven't fully thought it through but maybe this is where it should go. Appreciate the gov-data campaigning context too.

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#63

Post by jaggs »

mebkorea wrote: post #61
jaggs wrote: post #58

I would try and go open source as fast as possible before a legal letter land on your desk. Then worry about the commercialisation. Also I have a feeling you could charge SERIOUS coin for some app for property developers based around this. But someone is almost certainly going to come at you because, you know, us Brits hate clever clogs.

Yeah, unfortunately you are not wrong about the national tradition of wanting to pull down clever-clogs... The open source angle is something I'm increasingly considering, especially after a local government IT person made a fair point on this thread about the strain it causes. It won't fix the scraping load directly, but might frame the project as public-interest rather than attempting to make a bit of extra money.…

I think you're absolutely on the right track. It's all a matter of optics. Open source gives you the higher ground for the jobsworths. Meanwhile you can put together some kind of cool package to approach property developers with.

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#64

Post by mebkorea »

codeulike wrote: post #45

I'd be careful because even though its 'public' data, scraping it might not be legal due to TOS of the various sites. I did a search for my postcode and got given results for a different area and council miles away

All fixed. The bug was in how I was building the postcode-to-council mapping.

The script was extracting prefixes from individual application addresses, and Ceredigion's database happened to have a chunk of records with HR* postcodes (data entry errors at source. The addresses are Welsh places like Aberystwyth and Borth, but with Hereford postcodes attached). Those errors polluted the lookup table, so HR* ended up pointing at Ceredigion.

Fixed by trusting only the manual postcode-to-council mappings for councils that have them, rather than supplementing from address extraction. The postcode you sent now correctly shows "not yet in coverage" rather than results from rural Wales. Cheers for flagging. That was a zinger of a bug.

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#65

Post by [deleted] »

mebkorea wrote: post #64
codeulike wrote: post #45

I'd be careful because even though its 'public' data, scraping it might not be legal due to TOS of the various sites. I did a search for my postcode and got given results for a different area and council miles away

All fixed. The bug was in how I was building the postcode-to-council mapping. The script was extracting prefixes from individual application addresses, and Ceredigion's database happened to have a chunk of records with HR* postcodes (data entry errors at source. The addresses are Welsh places like Aberystwyth and Borth, but with Hereford postcodes attached). Those errors polluted the lookup table, so HR* ended up poi…

[deleted]

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#66

Post by mobilio »

mebkorea wrote: post #7
beatthatflight wrote: post #3

Worth trying claude/gemini to see if they'll do some scraping for you. I've found some paywall sites only too happy to allow Gemini past the wall.

Hadn't thought of that tbh. Worth a go on Liverpool especially... that's the AWS WAF one I'm currently blocked on and it is doing my head in. The challenge there is volume rather than access (~80k decisions to backfill), so even if an LLM gets through the wall I'd still need to script around it. But could be a way in for the initial cookie. Cheers for the tip and will look into it.

can you share link to Liverpool so i can try it too please?

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#67

Post by [deleted] »

jayelbe wrote: post #57

As somebody who works in local government IT, consistent scraping of our data like this is the bane of our life. We get hit by thousands of these, many with no rate limiting, making hugely intensive requests, that cause downtime and knock-on effects for actual customers and citizens. We block IPs, add captchas, and yet it persists. If you really want the data, just FOI it for goodness' sake. I get the distinct impres…

[deleted]

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#68

Post by lifeisstillgood »

mebkorea wrote: post #62
lifeisstillgood wrote: post #56

Earlier quoted context omitted.

I agree on the “public” data issue - I spent a long time campaigning for better FOSS / data access in government and there are some great people pushing in and outside local and central gov. But it’s a big mindset chnage (one that will benefit the whole country) but it’s slow. I think the “push for public policy improvements” angle if genuine will get you a lot more respect and kudos when things get sticky. Good luck

Cheers, and things get sticky isn't lost on me, especially after some of the feedback. Seems all the feedback is converging on the same direction namely open-source the data layer, lead with public-interest, let any commercial product sit on top rather than be the thing. Haven't fully thought it through but maybe this is where it should go. Appreciate the gov-data campaigning context too.

Yes … but…

No-one has figured out how to make money off open source (while sticking to the basic principles. Jeff Bezos makes a fortune off of it)

Most people who open source their code that I have known and still wanted to be paid / recognised for their effort have always been disappointed

Can I suggest you mentally put the work you have done to date into a box marked “the past”, open the data, start yourself as part of the community trying to make government code and data open, and sell your skills - the old “consultancy paying the bills” approach

Trying to make cash off public data will just confuse the message, and start to build resentment. Make a clean and clear statement, Sell your services on top. Expand to other forms of data scraping in government.

It’s a tough road - good luck

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#69

Post by mebkorea »

lifeisstillgood wrote: post #68
mebkorea wrote: post #62

Earlier quoted context omitted.

Cheers, and things get sticky isn't lost on me, especially after some of the feedback. Seems all the feedback is converging on the same direction namely open-source the data layer, lead with public-interest, let any commercial product sit on top rather than be the thing. Haven't fully thought it through but maybe this is where it should go. Appreciate the gov-data campaigning context too.

Yes … but… No-one has figured out how to make money off open source (while sticking to the basic principles. Jeff Bezos makes a fortune off of it) Most people who open source their code that I have known and still wanted to be paid / recognised for their effort have always been disappointed Can I suggest you mentally put the work you have done to date into a box marked “the past”, open the data, start yourself as par…

Thanks! Yup,... the freemium + paid analysis route is the obvious next step from where I am. You're right that my commercial intent is muddying the message. The sell skills, not data/ consultancy angle isn't one I'd seriously considered. Sounds pretty good, need to sit with it.

A few weeks ago, I'd have said the SaaS product is the play, but now I'm not so sure. Cheers for taking the time.

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#70

Post by mebkorea »

mobilio wrote: post #66
mebkorea wrote: post #7

Earlier quoted context omitted.

Hadn't thought of that tbh. Worth a go on Liverpool especially... that's the AWS WAF one I'm currently blocked on and it is doing my head in. The challenge there is volume rather than access (~80k decisions to backfill), so even if an LLM gets through the wall I'd still need to script around it. But could be a way in for the initial cookie. Cheers for the tip and will look into it.

can you share link to Liverpool so i can try it too please?

Sure, here you go: https://lar.liverpool.gov.uk/planning/index.html?fa=search

Heads up tho... it's behind AWS WAF with a JS challenge. Solving the challenge once works fine, but the WAF rate-limits the IP after ~10 requests and blocks for the rest of the day. So getting a session is doable, getting through 80,000 decisions is the hard bit. If you crack it I want to know! Cheers.

Post reply on HN