101–110 of 202 posts
Very interesting. I have written scrapers for the jail inmate data in the couple of counties nearest me - does that come under the scope of what you're doing, or not quite?
They're not cleaned up and ready for release, but I could do that if it's useful to the project.
At least a few more counties in my state use the same software.
Earlier quoted context omitted.
https://docs.pdap.io/activities/data-sources/explore-data-so... It's in the docs.
It still isn't very clear what the focus and direction they want to go in is from that page. What would be an example of a core data set they are trying to compile? police involved shootings? police budgets? everything.
> Our mission is to make data from every U.S. police agency accessible via a single public resource.
More precisely:
> There are over 18,000 police organizations, and each has a unique way to publish information.
i.e. Police are publishing data? Let's organize it and make it easily accessible.
So far this year, 177 LEO officers have died in the line of duty. Our gratitude should go to all.
There are other sources regarding Brady lists like https://giglio-bradylist.com/ and http://bradycops.org/, but they are obviously not 100% complete.
On the back end, are you using a graph? Having done some public sector accountability stuff where the org structures themselves were obfuscated, graphs and a clear data model were the decisive tech.
Can you say more about this? Feel free to reach out to my email (josh.chamberlain@pdap.io) if you'd like to share more. It sounds like you have some expertise that would be incredibly useful to us.
I have used it in different levels of govt to map managers to financial line items, to applications, corporate entities, projects, contract counterparties, platforms, techs, machines, ip addrs, vulnerabilities, etc. Developing a clear and addressable ontology of huge organizations with tens of thousands of people and devices is probably my one of my more useful skills. The main use case for graphs to me is patchy data, where you have a pile of incomplete metadata in dispirate spreadsheets and you need to find coherent paths through all of it.
I won't be in touch because I know what those people are capable of, but if graphs haven't accelerated your work already, you have some really epic times ahead!
Really love the idea, and the passion behind it. Def could have legs. Here’s the pitfalls I see you falling into: (1) seriously, what data are you collecting? “Everything” isn’t a great answer (who’s supposed to use ‘everything’, anyway? “Anyone”?). “Apples-to-apples police misconduct statistics” is a good one. (2) it’s important to clarify 1 because you need to know who you’re serving, and why. Different activists n…
Apologies for my ignorance but how is this going to police the police? I read the original blog post, there was lots of inferences/could and might be's/etc made but little in the way of proof of anything. What's to stop the police saying it was just circumstance that provided your results? I'm not here defending the police, or denigrating the project, just playing devils advocate. What happens if the police just igno…
Assuming the data is accurate it can be used to show disparities between groups for a variety of situations - traffic stops, arrests, jail vs. diversion programs, charge stacking, etc..
Earlier quoted context omitted.
Assuming the data is accurate it can be used to show disparities between groups for a variety of situations - traffic stops, arrests, jail vs. diversion programs, charge stacking, etc..
How is that policing the police? Are disparities supposed to be evidence of something nefarious going on? Given that there are fundamental distinctions between members in different groups (otherwise they would be in the same group) and almost certainly many other non-fundamental distinctions that correlate with the group-defining distinction, is it not entirely plausible that there should be disparities in police sta…
The National Crime Victimization Survey says yes. Also any article you see trying to debunk FBI crime stats but doesn't mention the NCVS (and how the NCVS largely corroborates the FBI stats) is either ignorant or willfully deceiving you.
Also relevant: So far this year, 177 LEO officers have died in the line of duty. Our gratitude should go to all. https://www.odmp.org/search/year/2022
For folks who do this kind of disparate data-source scraping at scale, what does best practices look like? What kind of tools are used in industry? Maintaining scrapers for 18k county websites and PDs is no small task and looking through the docs for PDAP, it seems like this is still a very open question.
The 80000 Hours podcast has an interview with the (non-technical) creator of OWID. I seem to recall some interesting stories about them getting emailed PDFs with COVID data and such.
I had the same question as you, and I was hoping to find ideas in the comments. It seems like the kind of thing that's both inherently messy and scrappy yet if you don't get at least somewhat organized it can't scale.
Update: link to the podcast episode page with quotes, transcripts, etc. https://80000hours.org/podcast/episodes/max-roser-our-world-...