Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scraping 241 UK council planning portals – 2.6M decisions so far

news.ycombinator.com

Post reply on HN
Print view View original

71–76 of 76 posts

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#71

Post by efaref »

ricardobayes wrote: post #19
efaref wrote: post #12

Great site. This data should really be more accessible. Planning in the UK is a total crapshoot, subject to the whims of the planning authorities. In our case, a simple rear extension and dormer loft conversion, similar to hundreds of thousands across the country, we ended up having to appeal which added 2 years and tens of thousands of pounds in costs to our extension project. Our area shows up as a high refusal are…

Maybe a tongue-in-cheek comment but regulations are that way because you guys want it that way (maybe not you personally). If it wasn't like that, nothing would stop a garbage incinerator or a quarry popping up a few hundred meters from houses (which happens in European countries with more deregulated planning/zoning regulations). You guys have all kinds of pro-individualistic, borderline nonsensical residental housi…

NIMBYism and BANANAs are certainly a problem, but this is the planning departments themselves acting capriciously , failing to even apply the law correctly, or having extremely onerous requirements. This applies to individual householders as well as large infrastructure projects. Just look at the lower thames crossing, where we spent £1.2 BILLION pounds of taxpayers money on pre-construction costs and compare that with the original Dartford tunnels and bridge which cost £191M TOTAL in 1990. It's just bonkers.

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#72

Post by ferngodfather »

mebkorea wrote: post #48
ferngodfather wrote: post #16

Your terms: > You may not use automated tools to scrape, copy, or bulk-download data from our service. Pot kettle, huh.

Updated and pushed live: planninglens.co.uk/terms. Acceptable Use clause now permits programmatic access that respects rate limits, while still protecting our derived analysis and reports. Thanks for the kick.

Fair play!

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#73

Post by stevem1978 »

I just signed up because of this thread. this is something I did 12 months ago. I only did northwest IDOX including Wirral and Liverpool. It was a long slog so i salute you.

I am not experienced at this at all but I managed to get around 107k from Wirral. the parsing afterwards was a bit crap. I could probably dig out the code from somewhere. the approach i took was quite long winded:

created a txt file with ID numbers 100000 to 200000 (or similar) then i think i used playwriter to pick a random number and scrape it then write the details to a CSV. then in another script i checked the CSV to see if the decision date was present and if it wasn't then repopulate the txt file. a massive pain and it of course changes every week!

I can think of a good use case and people who would be interested in this data though. I would love to see your work, especially for IDOX. don't give up, as others have said change your business model. members of the public are not your audience.

I dont think. look at selling the whole database rather than one line item. you will probably copies of the PDF decision notices etc. Oh and people who say that you should do a FOI.... ha! you would need to FOI every single house and street in the borough. the work that would create for the council would be expedential. The council charge businesses to access this free data. Councils should make the data available as a free download hosted. then all this would stop.

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#74

Post by PhilipV »

mebkorea wrote: post #51
lifeisstillgood wrote: post #49

Some thoughts 1. Brilliant! Governments (and corps) treat public data like it’s theirs not ours. Information yearns to be free. 2. Having said that, you are likely violating T&Cs by scraping at all. 3. It is a lot easier to defend your position if you are making it free and public yourself. 4. But paying for food is nice 5. I suggest the business model here is providing architects and lawyers with strong evidence of…

Cheers, appreciate the feedback. The architect/consultant precedent angle is interesting and a couple of other commenters have already nudged me in similar directions. Tbh... you're likely right that the strongest commercial play isn't B2C £19 reports, it's giving someone fighting a contested case the national pattern across 15 similar pub conversions, the appeal outcomes, what stuck and what didn't. That's a very di…

I would say that you should consider checking whether the councils and/or these planning portals operate under an Open Government Licence : https://en.wikipedia.org/wiki/Open_Government_Licence . These in theory permit commercial use.

I would also see that there is a strong case to say that it is aggregation. Something that would help is redirecting users to the original source content whenever you are referencing it within your search.

Re: Scraping 241 UK council planning portals – 2.6M decisions so far

#75

Post by pbhjpbhj »

mebkorea wrote: post #38
pbhjpbhj wrote: post #26

Is any of the data on Gov.uk - any scrapping tips there? I've tried scraping some patent tribunal data but haven't been successful (just using Python (copying in session data), I guess Playwright might be useful there).

Planning data on gov.uk is really patchy and not useful for what I want. There's planning.data.gov.uk which has some boundary/policy data but no actual decisions. The decisions only exist on council portals, which is the whole reason this project exists. On patent tribunal, I haven't looked into that one specifically but a few general gov.uk tips: most gov.uk content is actually clean HTML (way easier than council po…

Thanks for your help, I'll have a tinker with Playwright and see if I can get what I want.
Post reply on HN