I'm curious if there are opportunities to be a force multiplier here. I see that the Readme says "there's no automated scraper farm" yet. Getting that set up seems crucial. Will jump on the Discord :)
I accidentally started a movement – Policing the Police by scraping court data
11–20 of 202 posts
Re: I accidentally started a movement – Policing the Police by scraping court data
#12I'm curious if there are opportunities to be a force multiplier here. I see that the Readme says "there's no automated scraper farm" yet. Getting that set up seems crucial. Will jump on the Discord :)
The community has been writing scrapers since the beginning, but it's a huge task. We've also written scraper templates, since many counties use the same systems. https://github.com/Police-Data-Accessibility-Project/PDAP-Sc...
Re: I accidentally started a movement – Policing the Police by scraping court data
#13Thanks for sharing about your project!
Do you mind giving us brief on what kind of data you are collecting and highlight any interesting findings so far?
Re: I accidentally started a movement – Policing the Police by scraping court data
#14I like writing web scrapers and this is an interesting project idea. If I understand right you are looking for volunteers to write scrapers that would take a police department, scrape the PD website, and download any PDFs or documents that gather data about the police department. Is that right? If so, I feel that's not super clearly communicated - I had to look at a couple example scrapers before arriving at this guess.
I do have a few questions too:
1. Will this scale? One problem with scrapers is that they break when people update their website. I'm imagining this problem multiplied by 18,000 and compounded by each scraper potentially being written by a different volunteer.
2. Where are the scrapers getting run?
3. How do the documents that the scrapers collect get transformed into usable data?
4. It seems to me like a scalable solution would be a standard to report data, a law to compel police departments to follow that standard, and then a system to collect that data and make it available. Do you work with police departments at all on data reporting?
Re: I accidentally started a movement – Policing the Police by scraping court data
#15I'm curious if there are opportunities to be a force multiplier here. I see that the Readme says "there's no automated scraper farm" yet. Getting that set up seems crucial. Will jump on the Discord :)
The community has been writing scrapers since the beginning, but it's a huge task. We've also written scraper templates, since many counties use the same systems. https://github.com/Police-Data-Accessibility-Project/PDAP-Sc...
Yep, I see that there are a lot of existing scrapers. Getting those up and running automatically on a backend host will make things a lot smoother.
Also, I recommend using Playwright instead of Selenium. I'm in the Discord now and will be hanging out, so looking forward to chatting more about this and contributing.
Re: I accidentally started a movement – Policing the Police by scraping court data
#16No post body was provided.
Re: I accidentally started a movement – Policing the Police by scraping court data
#17a while back I created www.bartcrimes.com to publish police reports which were intentionally hidden behind a mailing list you must get approved to be a member of. Turns out, the public loves this kind of thing.
Re: I accidentally started a movement – Policing the Police by scraping court data
#18Quoted post unavailable.
No post body was provided.
Re: I accidentally started a movement – Policing the Police by scraping court data
#19Quoted post unavailable.
could you elaborate exactly where the filth is here? I think I get it, but I'm not super-well versed enough to understand what the motive would be here.
Re: I accidentally started a movement – Policing the Police by scraping court data
#20Quoted post unavailable.
The post you quoted lists potential avenues for using web3 tools. You may notice we have not gone down any of those avenues. We aren't doing crypto stuff.
We did indeed spend some time looking into web3 and peer-to-peer, and ultimately decided it's not for us. Most of the energy there is spent trying to make money / scam people. Peer-to-peer is cool too but has its own risks.
Thanks for checking out the discord, though!