Live data from Hacker News

Using FOIA Data and Unix to halve major source of parking tickets

mchap.io

11–20 of 248 posts

Re: Using FOIA Data and Unix to halve major source of parking tickets

#11
post #8

Earlier quoted context omitted.

You could have these hobbies!

I'm kind of on a negative streak these days, so forgive me, reader, but...why in god's green earth aren't these the government's hobbies?!

You’re not wrong, not in the slightest. The benefit is the system allows concerned citizens to step in to make improvements at very little cost.

Democracy requires constant vigilance in many forms. This is one such form.

Matt, we salute you and your efforts! I hope this encourages others to get involved in improving their local government (and perhaps even creates reusable tooling for use at scale [“citizen oversight as code”]).

Re: Using FOIA Data and Unix to halve major source of parking tickets

#12
post #9

This is awesome. Question though: how is producing license plate data like this not a disallowed privacy invasion? It seems like you could totally track who's parking where and potentially do nasty stuff, if you know (say) someone well-off whom you don't like and who doesn't seem to mind getting tickets on a regular basis.

Criminal proceedings are usually public in most countries. Where do parking tickets fall? Even if not technically criminal, some might, for the same underlying reasons, consider it acceptable for "someone well-off whom you don't like and who doesn't seem to mind getting tickets on a regular basis" to have this illegal behavior on the public record. "Don't want your name tarnished? Don't park illegally."

[Edit: on the other hand, if the ticket is unfair (eg. confusing signage as in this example), then you have a valid point; I just wanted to point out the other side of the coin]

Re: Using FOIA Data and Unix to halve major source of parking tickets

#14

Earlier quoted context omitted.

He also spends a lot of time corresponding the old fashioned way to get the data & cleaning it for mechanical evaluation. His hobbies make me feel bad about my own...

You could have these hobbies!

That's what I'm saying!

Re: Using FOIA Data and Unix to halve major source of parking tickets

#16
post #9

This is awesome. Question though: how is producing license plate data like this not a disallowed privacy invasion? It seems like you could totally track who's parking where and potentially do nasty stuff, if you know (say) someone well-off whom you don't like and who doesn't seem to mind getting tickets on a regular basis.

The TL;DR is: "any criminal records is public information". Fundamentally, even if it's very minor, it's public information until expunged, cleared, or sealed. I think most people are generally fine with this, it's only really a problem if you consistently are committing offenses.

Re: Using FOIA Data and Unix to halve major source of parking tickets

#17
Reminds me of working with free-form, manual entry order detail information, in a former life.

Hundreds of thousands of records a month. I ended up importing them into Excel(1) and then using... what was that called? An MS/Windows library that came with IE 5 and/or a few other things, that provided regex support (with a few quirks) that was accessible via VBA.

The point was, I could programmatically mine it -- including regex pattern matching and replacement of and within cell contents -- while also having a flexible UI within which to find and handle one-off cases. When the one-off's demonstrated a repeating pattern, I could quickly iterate to add that to the programmatic mining logic.

This included adding color cueing for items of particular interest, manual follow-up. Excel's sorting capabilities to bring potentially related instances into visually displayed groups. And the like.

It ended up working quite well. I might have preferred something else to VBA, and I did use Perl and other stuff, elsewhere (something that also gave me both power and the flexibility to rapidly iterate).

But the point is, with such data, I found it very useful to combine regex and rapid programmatic manipulation, together with a good visual interface (including visual cues, the ability to comment upon instances -- Excel cell-level comments -- etc.) and manual manipulation.

As a final aside, the extensive set of Excel keyboard shortcuts greatly aided in rapidly and effectively navigating and massaging the imported data.

--

1. This was back when Excel had... I think it was a 64K (or a bit less) limit on the number of rows in a sheet.

P.S. I tended to retain the originally imported data in its columns, and to produce my mining of it in other columns. That way, I could always and immediately see what I started with, for any particular record. (And, if things visually started to be "too many columns", well, Excel lets you hide a range of columns from the view. As one example of how its features really helped, on the visual front while doing this work.)

I still had to learn and allow for some quirks Excel exhibited with respect to importing text data. That included making sure the cells/columns being imported into carried the correct/needed formatting designation before importing into them (usually, "Text").

Re: Using FOIA Data and Unix to halve major source of parking tickets

#18
post #9

This is awesome. Question though: how is producing license plate data like this not a disallowed privacy invasion? It seems like you could totally track who's parking where and potentially do nasty stuff, if you know (say) someone well-off whom you don't like and who doesn't seem to mind getting tickets on a regular basis.

In the US there is an entire industry that scans license plates on public streets and sells the data. Basically no privacy in public.

(random example: https://www.syracuse.com/news/index.ssf/2015/01/private_comp... )

Re: Using FOIA Data and Unix to halve major source of parking tickets

#19
post #8

Earlier quoted context omitted.

You could have these hobbies!

I'm kind of on a negative streak these days, so forgive me, reader, but...why in god's green earth aren't these the government's hobbies?!

Because different people have different priorities.

Remember, you're part of the government.

Re: Using FOIA Data and Unix to halve major source of parking tickets

#20
post #12
post #9

This is awesome. Question though: how is producing license plate data like this not a disallowed privacy invasion? It seems like you could totally track who's parking where and potentially do nasty stuff, if you know (say) someone well-off whom you don't like and who doesn't seem to mind getting tickets on a regular basis.

Criminal proceedings are usually public in most countries. Where do parking tickets fall? Even if not technically criminal, some might, for the same underlying reasons, consider it acceptable for "someone well-off whom you don't like and who doesn't seem to mind getting tickets on a regular basis" to have this illegal behavior on the public record. "Don't want your name tarnished? Don't park illegally." [Edit: on the…

What you say is true. It's also the case that there's a lot of information that is nominally public--often for very defensible reasons--which historically took significant effort to get at and, even then, only in a very limited and manual way.

Today, a lot of that information is at least a lot more accessible to everyone (though in this case it still took a lot of work) and, furthermore, it can be mashed up with other public or semi-public data.

I'm pretty sure this is something we'll be collectively be coming to terms with for a long time.

Post reply on HN