Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I accidentally started a movement – Policing the Police by scraping court data

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 202 posts

Re: I accidentally started a movement – Policing the Police by scraping court data

#121

Post by josh-pdap »

lazyasciiart wrote: post #72
account-5 wrote: post #71

Apologies for my ignorance but how is this going to police the police? I read the original blog post, there was lots of inferences/could and might be's/etc made but little in the way of proof of anything. What's to stop the police saying it was just circumstance that provided your results? I'm not here defending the police, or denigrating the project, just playing devils advocate. What happens if the police just igno…

The goal of projects like this (I have no contact with this one) is usually to convince politicians and/or the public of their results, and those groups are the ones to actually push change.

In our case it’s about working with local activists and journalists. Police are decentralized; each department acts independently, for the most part. Most people using the data seem to be using it locally; when policies are changed, they are changed locally. If the changes make a positive impact, sometimes other departments or governing bodies take notice.

I’d like to aim higher than a quotable statistic for a politician, in any case :)

Re: I accidentally started a movement – Policing the Police by scraping court data

#122

Post by josh-pdap »

elicash wrote: post #7

I'd be interested in helping scrape, but no experience. I'd presume every county is different so there's no simple training you can put folks through? Other tasks, like monitoring for things breaking?

Thanks for wanting to help! If you go to https://docs.pdap.io you should be able to find out how to contribute Data Sources. No coding required. Holler in Discord or email me (josh.chamberlain@pdap.io) if you have trouble!

Re: I accidentally started a movement – Policing the Police by scraping court data

#123

Post by josh-pdap »

motohagiography wrote: post #105
josh-pdap wrote: post #76

Earlier quoted context omitted.

Can you say more about this? Feel free to reach out to my email (josh.chamberlain@pdap.io) if you'd like to share more. It sounds like you have some expertise that would be incredibly useful to us.

Thanks! Work I did was with Neo4j, which a reasonably technical person can learn in a few days using their movie-db tutorials, where once you have a clear idea of what the categories are in your data and what is a top level "thing" vs. what an attribute of a thing is, you can get correlations. (persons, orgs, parties, IP addrs, and maybe events are the things, where addresses, dates, operating systems, contact info,…

I appreciate you taking the time to explain this to me. As a non-engineer I think I’m following you.

When you say “I know what those people are capable of” do you mean something as ominous as that sounds?

Re: I accidentally started a movement – Policing the Police by scraping court data

#124

Post by josh-pdap »

notacop31337 wrote: post #93
contingencies wrote: post #28

I wonder if it is legal / possible to record police radio traffic and associate it with the records?

Depending on the country it can be varying levels of "illegal", in Australia, Police use encrypted radio devices, P{01-99} Listening is not illegal, but recording or redistributing "is illegal", however, it's not clear whether it's actually illegal or it just depends how and when you use it and what you do with it. There was a kid here that brought it up with the police and was harassed over it, I believe he made a w…

We do want to be careful with PII. We don’t want to make a big table of PII, although tables of PII are regularly published to government websites as html tables. Yet another reason to keep data decentralized while pointing people to it from a central db of sources!

Re: I accidentally started a movement – Policing the Police by scraping court data

#125

Post by josh-pdap »

KennyBlanken wrote: post #89

Are you also working on pushing standards for data sources, such as a state-level standard? Ideally federal standards? Maintaining thousands of scrapers for different formats seems like a nightmare, and it won't take long for departments to learn they can slightly tweak the format of their reporting to cause extra work for you. On the plus side, working with all this data probably makes you all very qualified to advi…

We definitely want scrapers to be maintained as-needed, as opposed to trying to spin all the plates. It’d be thousands of thousands.

Measures for Justice is working on developing standards: https://measuresforjustice.org/

Re: I accidentally started a movement – Policing the Police by scraping court data

#126

Post by josh-pdap »

lazyasciiart wrote: post #69

Very interesting. I have written scrapers for the jail inmate data in the couple of counties nearest me - does that come under the scope of what you're doing, or not quite?

Yes! Police, jails, and courts. Those are all under the umbrella of criminal justice data, and all contextualize each other.

Could you share what you’ve made somehow? josh.chamberlain@pdap.io or discord are likely the best way.

Would also be curious to hear what made you take this on.

Re: I accidentally started a movement – Policing the Police by scraping court data

#127

Post by josh-pdap »

mod wrote: post #101
lazyasciiart wrote: post #69

Very interesting. I have written scrapers for the jail inmate data in the couple of counties nearest me - does that come under the scope of what you're doing, or not quite?

I also have inmate scrapers running (and republishing the data on a better website). Mine apply to any county using a specific software. They're not cleaned up and ready for release, but I could do that if it's useful to the project. At least a few more counties in my state use the same software.

Interesting, what are you using them for?

Most of our scrapers are written by inexperienced volunteers so this is probably still quite valuable as is. If it works, we’ll take it.

You could also raise a hand as a human data source; perhaps one day someone will be looking for jail data from these counties and you could offer up some way to contact you. I gave my contact info to the person you replied to. Use it if you want!

Re: I accidentally started a movement – Policing the Police by scraping court data

#128

Post by josh-pdap »

kylewatson wrote: post #78
kristintynski wrote: post #4

Earlier quoted context omitted.

You can find the roadmap, goals, and a lot of detail on the organizational structure and the scrapers that have been written so far are here https://docs.pdap.io/activities/our-process and here https://github.com/Police-Data-Accessibility-Project

With all due respect, you really need to be able to explain the goals in a sentence or two, and then follow up with a link to the pdf. An elevator pitch, a vision statement, a clearly written user statement, all would go a long ways towards explaining why this is a worth while use of ones time and energy. I don't know whether the goal is to find bad cops who have been fired and get hired one town over and try to prev…

Thanks for all this. Articulating why it’s important in a few words is incredibly difficult and something we’re always working on. The bottom line is that we believe public information should be easily accessible; hundreds of thousands of people are using this data currently, all over the country. We’re trying to make their lives easier. I’ll take your feedback and use it to make our stuff better.

That said, if you head to https://pdap.io you’ll find the most concrete explanation we have. It still needs to be clearer, but it’s more specific than the docs.

Re: I accidentally started a movement – Policing the Police by scraping court data

#129

Post by josh-pdap »

RickJWagner wrote: post #103

Also relevant: So far this year, 177 LEO officers have died in the line of duty. Our gratitude should go to all. https://www.odmp.org/search/year/2022

The FBI and other police organizations are certainly interested in using police data to make the job as safe as possible, for the police as well as the people they’re sworn to protect and serve.

Re: I accidentally started a movement – Policing the Police by scraping court data

#130

Post by celestialcheese »

electromech wrote: post #110
celestialcheese wrote: post #82

For folks who do this kind of disparate data-source scraping at scale, what does best practices look like? What kind of tools are used in industry? Maintaining scrapers for 18k county websites and PDs is no small task and looking through the docs for PDAP, it seems like this is still a very open question.

Our World In Data is the largest open source data collection & analysis that I'm aware of. https://github.com/owid The 80000 Hours podcast has an interview with the (non-technical) creator of OWID. I seem to recall some interesting stories about them getting emailed PDFs with COVID data and such. I had the same question as you, and I was hoping to find ideas in the comments. It seems like the kind of thing that's bot…

Thanks for sharing!

It's interesting that even one of the largest still uses manual execution for almost all of their pipelines (at least in the covid data project[1]). This [2] seems like the bulk of their data importers (scrapers) but most are still operating as manual jobs. I guess with open-source data work, hours and minutes don't matter as much, and being a few days behind the latest data is acceptable.

1 - https://docs.owid.io/projects/covid/en/latest/data-pipeline.... 2 - https://github.com/owid/importers

Post reply on HN