Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I accidentally started a movement – Policing the Police by scraping court data

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 202 posts

Re: I accidentally started a movement – Policing the Police by scraping court data

#61

Post by anamexis »

josh-pdap wrote: post #58
Pelerin wrote: post #48

Thank you for your work with this! One question I have: You say in your FAQ "We aren't a watchdog—our activism is data collection and accessibility, not analysis or research." Can you note any instances of other people using your data for analysis or research?

We're still developing those relationships, and we haven't generated any novel data that is deeper than web URLs. I'm based in Pittsburgh so we're still working with local journalists, activists, etc. to understand how they use the data and how we can help.

Just a heads up that your comments are getting marked as dead (shadow banned). You seem to have fallen afoul of some HN spam trap for new accounts.

Re: I accidentally started a movement – Policing the Police by scraping court data

#63

Post by steve76 »

kennywinker wrote: post #38
RobertRoberts wrote: post #35

Is their any data sources we could scrape to stop crimes in our neighborhoods so the police don't have any reason to come around and cause problems?

You could scrape improved social safety nets, better access to birth control, and taxing the rich to pay for it.

No post body was provided.

Re: I accidentally started a movement – Policing the Police by scraping court data

#64

Post by VWWHFSfQ »

josh-pdap wrote: post #56
VWWHFSfQ wrote: post #43

Is it possible to see the data the PDAP has scraped? I visited the website but I don't see any actual data.

We don't have any scraped data yet. I replied to the parent post addressing some of this, but mostly if people need the data they run a scraper locally and use the data that way. At the moment, our energy is going into building an app to help people submit and manage our database of data sources: https://docs.pdap.io/activities/data-sources/what-is-a-data-...

Ah very good! This sounds like it would be a perfect application of https://datasette.io/

Re: I accidentally started a movement – Policing the Police by scraping court data

#65

Post by soco »

icelancer wrote: post #50
josh-pdap wrote: post #44

Hello! I'm the executive director. I have a design background, have done product management in the past, and aside from keeping the lights on at PDAP and making sure we're tax-compliant I am in a product role. I talk to people using police data, and figure out where we can add value to make the data more accessible. TL:DR; If you want to write scrapers: go for it! Run your scraper, share the results in Discord and wi…

Not sure why this got killed (dead in HN terms) but I vouched for it.

Half of josh's comments show up as dead, so maybe you and others who can, look around and vouch for them as well. There's something nefarious going on...

Re: I accidentally started a movement – Policing the Police by scraping court data

#66

Post by sophacles »

icelancer wrote: post #50
josh-pdap wrote: post #44

Hello! I'm the executive director. I have a design background, have done product management in the past, and aside from keeping the lights on at PDAP and making sure we're tax-compliant I am in a product role. I talk to people using police data, and figure out where we can add value to make the data more accessible. TL:DR; If you want to write scrapers: go for it! Run your scraper, share the results in Discord and wi…

Not sure why this got killed (dead in HN terms) but I vouched for it.

Over half this person's comments (the Josh-pdap person I mean) have been flagged.

Not sure if this is some false positive on anti-spam (new account, lots of comments in a short time on a single story) or if it's someone with a vendetta. @dang this is a little weird...

(I know, this site doesn't have magic @, but also i think the site moderator probably has scripts searching for mentions/summons anyway. I would anyway...)

Re: I accidentally started a movement – Policing the Police by scraping court data

#67

Post by Merciernmon »

josh-pdap wrote: post #44

Hello! I'm the executive director. I have a design background, have done product management in the past, and aside from keeping the lights on at PDAP and making sure we're tax-compliant I am in a product role. I talk to people using police data, and figure out where we can add value to make the data more accessible. TL:DR; If you want to write scrapers: go for it! Run your scraper, share the results in Discord and wi…

Josh, many of your comments here are displaying as "dead." The HN FAQ[0] says:

> What does [dead] mean?

> The post was killed by software, user flags, or moderators. Dead posts aren't displayed by default.

I suspect that it is a false positive. Maybe email the mods for help? hn@ycombinator.com

[0] https://news.ycombinator.com/newsfaq.html

Re: I accidentally started a movement – Policing the Police by scraping court data

#68

Post by josh-pdap »

jrockway wrote: post #59
icelancer wrote: post #50

Earlier quoted context omitted.

Not sure why this got killed (dead in HN terms) but I vouched for it.

Yeah, all of the new comments from the account are dead. I think creating a new account, posting an article, and posting a handful of comments looks a lot like spam to HN. I've vouched for the comments, but would definitely recommend emailing the admins. Josh, I can't reply to you directly, but yeah... you can see the comments, but most people can't, unless they have enabled "showdead" in their preferences.

Thanks! It wasn't me who posted the article, but I agree with the theory that it's likely some well-meaning "new account spam prevention" thing.

Re: I accidentally started a movement – Policing the Police by scraping court data

#70

Post by ALittleLight »

josh-pdap wrote: post #54
ALittleLight wrote: post #14

I like writing web scrapers and this is an interesting project idea. If I understand right you are looking for volunteers to write scrapers that would take a police department, scrape the PD website, and download any PDFs or documents that gather data about the police department. Is that right? If so, I feel that's not super clearly communicated - I had to look at a couple example scrapers before arriving at this gue…

1. I replied to the parent comment here; our answer to the scale problem is to recognize that people doing web scraping are as decentralized as the police. Our goal is to empower people who have questions about the police to answer them. 2. You can run them locally. We're not running the scrapers anywhere, or storing extractions anywhere. 3. This is a big, big question. Right now, the answer is dependent on the use c…

Not to be too rude/negative/mean - but it seems like a big concern to me that a "police data" project that's existed for three years, has a couple employees, has some funding, etc - doesn't really seem to have any police data. If I wanted to write a scraper to gather documents off of police websites and run it by myself and store my results locally - I could. What does your project add?

What I would expect to see is something like:

1. Here's what data we want from each police department each day. Here's what value you should use to indicate that data is not available.

2. Here's a list of police departments. Write a scraper. If it passes tests to show it's generating valid data, and code review, we will run the scraper in a daily basis storing the output in this database.

3. Here's how you can query our database.

Post reply on HN