Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The City of Seattle accidentally gave me 32M emails for $40 (2018)

mchap.io

Post reply on HN
Print view View original

51–60 of 230 posts

Re: The City of Seattle accidentally gave me 32M emails for $40 (2018)

#51

Post by kelnos »

ta1243 wrote: post #15
bradley13 wrote: post #11

Government IT is famously expensive, and so often a disaster. I recently needed to open an account with a local agency. The fun started with two date fields on the web form, which turned out (trial and error) to require different formats. I finally received my credentials, but they didn't work. Assuming that the password was likely the problem, I tried the password reset function, only to get a 404 error. The person…

The government probably has outsourced this to the only company which bid based on some "anti-corruption" or "fair-trading" rules which were passed. There is of course no way to hold that outsourced company to account.

More like the company made the lowest bid, and their bid was intentionally lower than the actual cost of building what the agency wanted built, so they did the least work possible to get it to vaguely work, and moved on.

Re: The City of Seattle accidentally gave me 32M emails for $40 (2018)

#52

Post by pvg »

refulgentis wrote: post #30
pvg wrote: post #26

Earlier quoted context omitted.

The author's email is at the bottom of their post if you need to reach them to tell them off about their failings from 5 years ago.

Its the author, and idk if the snide extra bit about informing them about their failings from 5 years ago makes any sense. This was posted today.

The email has the advantage of being unconstrained by various HN rules about not being a jerk to internet strangers.

Re: The City of Seattle accidentally gave me 32M emails for $40 (2018)

#53

Post by kelnos »

oh_sigh wrote: post #12

What happens if he refused the lawyer request to scan his hard drive?

He did refuse it, and nothing bad happened, it seems. Unfortunately, he had to get his own lawyer involved, who likely negotiated with the city's lawyer in order to get them to back off. That kind of assistance was probably not cheap for OP.

Re: The City of Seattle accidentally gave me 32M emails for $40 (2018)

#54

Post by legitster »

flashback2199 wrote: post #46
legitster wrote: post #42

Earlier quoted context omitted.

The dataset was so large they were probably not opening up the files before sending them over. It was clearly a rookie mistake by someone who had never had to do this stuff before and wouldn't have known what to look for. Again, the vast majority of the emails were automated alert/spam emails. So it's unclear if a random sample of the data would have turned up anything interesting to look at if you didn't know what y…

I don't agree, I think opening any of the files would have made it immediately apparent that the first 256 chars of each email were included, if that's indeed what happened as the article said.

>Some things about the dataset: It’s very messy – triple quotes, semicolons, commas, oh my. There are a millions of systems alerts. For seattle.gov → seattle.gov communication, there are two distinct metadata records

Opening a file of that size in Excel would probably crash the desktop. And this was probably a lowly admin without a lot of other tools.

The first 256 char of a lot of system emails are going to just have junk html and header tags. If you don't understand that you are looking at HTML, it's not going to be apparent that you are looking at the body of an email.

It's a rookie mistake to be sure, but the admin was clearly unfamiliar with what was being requested.

Re: The City of Seattle accidentally gave me 32M emails for $40 (2018)

#55

Post by kelnos »

legitster wrote: post #42
flashback2199 wrote: post #31

Earlier quoted context omitted.

> The job seems awful How hard can it be to do a sanity check of the metadata?

The dataset was so large they were probably not opening up the files before sending them over. It was clearly a rookie mistake by someone who had never had to do this stuff before and wouldn't have known what to look for. Again, the vast majority of the emails were automated alert/spam emails. So it's unclear if a random sample of the data would have turned up anything interesting to look at if you didn't know what y…

That's not a "rookie mistake"; that's gross negligence. Whenever I write some sort of script to produce some sort of data file, I always look to ensure that the data file ends up looking like what I expect.

This isn't even to ensure my data file doesn't include something private, just to ensure that it actually includes what I intended it to, and I didn't do something dumb like put the data in the same field twice, or duplicate the same record over and over, or whatever.

Re: The City of Seattle accidentally gave me 32M emails for $40 (2018)

#56

Post by legitster »

techsupporter wrote: post #20
nostromo wrote: post #8

Earlier quoted context omitted.

In the US emails to and from government workers are generally considered public record. It's important for transparency.

I agree that government records should be open and are good for transparency. The problem comes when people don't realize that something they're doing would put all of the personal information they've submitted into the public record, or when it's required to do this to access a service. In Washington, and Seattle specifically, doing something as simple as reserving a park space or signing up for a constituent newsle…

We already do this and put up with it with mailing addresses. The only difference is that the transaction cost of your email address is basically nil compared to your mailing address.

Re: The City of Seattle accidentally gave me 32M emails for $40 (2018)

#57

Post by kelnos »

legitster wrote: post #54
flashback2199 wrote: post #46

Earlier quoted context omitted.

I don't agree, I think opening any of the files would have made it immediately apparent that the first 256 chars of each email were included, if that's indeed what happened as the article said.

>Some things about the dataset: It’s very messy – triple quotes, semicolons, commas, oh my. There are a millions of systems alerts. For seattle.gov → seattle.gov communication, there are two distinct metadata records Opening a file of that size in Excel would probably crash the desktop. And this was probably a lowly admin without a lot of other tools. The first 256 char of a lot of system emails are going to just hav…

Would it really have been a XLS file? I would expect probably CSV? In that case, just running 'less' on the file (or the Microsoft equivalent) would be fine, and wouldn't tax anyone's desktop resources.

Re: The City of Seattle accidentally gave me 32M emails for $40 (2018)

#58

Post by datadrivenangel »

chankstein38 wrote: post #36
chaps wrote: post #2

Hah, that's a much simpler title. :)

How do I find more information out about FOIA requests? Also, in your post I think you mentioned "FOIA junkies" or something along those lines, this implies to me there's a community of people who understand this stuff and talk about it. That sounds really interesting to join. Is there a subreddit or something?

National Freedom of Information Coalition or Foia.wiki are great places to learn more.

https://www.nfoic.org/

Re: The City of Seattle accidentally gave me 32M emails for $40 (2018)

#59

Post by andersrs »

Companies and governments will always undermine privacy either by incompetence or because it's good for business. In every case the only party that benefits from more privacy is really the end user. So what can we as users do about this? Is there an email service that can generate unique alias addresses? just.for.bloated.govt.dept.123@xyz.com would really help but I'm guessing the big companies like Google and Microsoft have no interest in privacy.

Re: The City of Seattle accidentally gave me 32M emails for $40 (2018)

#60

Post by ta1243 »

kelnos wrote: post #51
ta1243 wrote: post #15

Earlier quoted context omitted.

The government probably has outsourced this to the only company which bid based on some "anti-corruption" or "fair-trading" rules which were passed. There is of course no way to hold that outsourced company to account.

More like the company made the lowest bid, and their bid was intentionally lower than the actual cost of building what the agency wanted built, so they did the least work possible to get it to vaguely work, and moved on.

...and internal rules will mean that the lowest bid will have to be taken.

Probably a bit of both. I'd be very nervous about jumping through the hoops that come with supplying to government -- hoops the people working in the government don't want, but ones that have been put in by politicians for various reasons.

Post reply on HN