I accidentally started a movement – Policing the Police by scraping court data

To best mimic the success of the original post, it would be good to be able to identify people doing the most damage to a neighborhood so we could help them first with tax relief, food, birth control, whatever would help the most.

TL:DR; If you want to write scrapers: go for it! Run your scraper, share the results in Discord and with your friends, and talk about the process. We'll be listening, and it will help us build tools to support this important work.

A few things to clarify:

a. The source of truth for "what are we doing right now" and "how can I contribute" is https://docs.pdap.io/.

b. Empowering people who write scrapers is a part of our broad mission of "police data accessibility", but we have some foundational work to do first! Right now our primary project is creating a database of police agencies and data sources. This will help people understand what kinds of data are available, at which agencies, with which steps to access it. It will also help us create archives of the primary sources, so that if they get taken offline we can still go back and scrape them.

c. What we have realized in the past few years: there are already a ton of people writing and using web scrapers for their day to day work. They are as decentralized as our police system. Our scrapers repo will reflect that. We shouldn't all rely on one library, or even one language. The people who need the data are most motivated to maintain scrapers, and we expect that maintenance will be ad-hoc and as-needed for the immediate future. In most cases, data already published on the internet is useful to local users as-is.

d. If you have a question you'd like to answer about the police, here's the investigation process:

1. Determine whether public data exists to answer your question. Use google to find the appropriate agency, and see what they're publishing. 2. Determine how it can be accessed; do you need to make a FOIA request? Is there a URL? 3. If there's a URL, determine whether you need to write a scraper to access the records. Often, the records can simply be downloaded. 4. Write and run a scraper, if you need one! 5. If there's not a URL, make a records request for the public information. This is a long and complicated process. 6. Share the data with your friends.

This means that scrapers are helpful and necessary some of the time; but not always, and not as the first step. We're trying to help with steps 1, 2, 3, 5, and 6. The theory is that writing scrapers is something people can easily slot in and help with; and that, depending on what question you're trying to answer, two scrapers for the same data source might look wildly different.

Scrapers are an important part of the ecosystem, but they're one piece of the puzzle.

You say in your FAQ "We aren't a watchdog—our activism is data collection and accessibility, not analysis or research."

Can you note any instances of other people using your data for analysis or research?