Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How MTA shut down my app for Penn Station commuters

medium.com

Post reply on HN
Print view View original

21–30 of 120 posts

Re: How MTA shut down my app for Penn Station commuters

#21

Post by Sir_Cmpwn »

falcolas wrote: post #9

Scraping the MTA site against the TOS, and intentionally obfuscating requests to limit the chance of the MTA identifying and shutting it down? Yeah, no surprises here. EDIT: Oops, missed that he didn't end up going live with the scraping. It would still be great for the MTA to publish this data intentionally; but I'm sure the potential for trains to switch tracks and the associated backlash when they do is what preve…

The author didn't do that, though. As far as MTA is concerned the app was always using an API he had been given a legitimate key for.

As an aside, fuck websites that prohibit scraping. If you send me some bytes I'm gonna do whatever I damn well please with them.

Re: How MTA shut down my app for Penn Station commuters

#22

Post by Sir_Cmpwn »

karrotwaltz wrote: post #11
drinchev wrote: post #3

Why the OP did not continue with scraping the website then? I think there will be no legal issue if he does that.

> They also stopped publishing the track number on the website — so going back to scraping was not an option either. So the only way to gather the data would be manually from the station.

Carefully placed raspberry pi + OCR?

Re: How MTA shut down my app for Penn Station commuters

#23

Post by EGreg »

Sir_Cmpwn wrote: post #21
falcolas wrote: post #9

Scraping the MTA site against the TOS, and intentionally obfuscating requests to limit the chance of the MTA identifying and shutting it down? Yeah, no surprises here. EDIT: Oops, missed that he didn't end up going live with the scraping. It would still be great for the MTA to publish this data intentionally; but I'm sure the potential for trains to switch tracks and the associated backlash when they do is what preve…

The author didn't do that, though. As far as MTA is concerned the app was always using an API he had been given a legitimate key for. As an aside, fuck websites that prohibit scraping. If you send me some bytes I'm gonna do whatever I damn well please with them.

In that case screw your API, if you allow DDOS I may as well do it for the lulz.

Isn't it the same approach? If you can abuse a site you will, because they opened themselves up to the public?

Re: How MTA shut down my app for Penn Station commuters

#24

Post by CPLX »

state_machine wrote: post #4

I can see why the MTA wouldn't want you telling people to go to a track before it is actually assigned -- if the train ends up on a different track, now you need to get all those people back up, off that platform an on to the new one, clogging stairs that that people who actually wanted that platform might be trying to use too. Obviously it would be nice to assign tracks earlier, so that people can head straight to t…

It's more serious than that even, it's a safety issue. The platforms are fairly narrow and have no railings, and they aren't designed to hold two full train loads of people at the same time.

If a full train pulls up and the platform is already full people are going to end up on the tracks. I haven't seen this at Penn but I have seen the issue happen in the subway when there are train delays. When this happens the MTA has to hold the arriving train in the station with the doors closed and clear the platform before opening the doors to let people off.

It's a nightmare. Their concerns here seem completely reasonable.

Re: How MTA shut down my app for Penn Station commuters

#26

Post by tedmiston »

CPLX wrote: post #24
state_machine wrote: post #4

I can see why the MTA wouldn't want you telling people to go to a track before it is actually assigned -- if the train ends up on a different track, now you need to get all those people back up, off that platform an on to the new one, clogging stairs that that people who actually wanted that platform might be trying to use too. Obviously it would be nice to assign tracks earlier, so that people can head straight to t…

It's more serious than that even, it's a safety issue. The platforms are fairly narrow and have no railings, and they aren't designed to hold two full train loads of people at the same time. If a full train pulls up and the platform is already full people are going to end up on the tracks. I haven't seen this at Penn but I have seen the issue happen in the subway when there are train delays. When this happens the MTA…

> The platforms are fairly narrow and have no railings, and they aren't designed to hold two full train loads of people at the same time.

It seems pretty unlikely that 100% of the people on the outbound train would use the app and trust the historical data to go to the platform early.

Re: How MTA shut down my app for Penn Station commuters

#27

Post by mjmasn »

dmurray wrote: post #8
mjmasn wrote: post #2

Would have been interesting to see if you could reliably crowd-source the data from commuters themselves.

Google almost certainly already could, without even asking the commuters, since they have real time location for so many Android phones. But there's a chicken and egg problem: it's hard to get the data before you're able to offer anything in return.

Chicken and egg yes, but he had the recent historical data and a bunch of paid-up users.

I'm just amazed this app was even necessary in a first-world transport system. For all their faults, the UK rail network is pretty good at providing timely information (and as some other comments have mentioned have solid APIs for this stuff). Same with the LTA[1] in Singapore.

[1] https://www.mytransport.sg/content/mytransport/home/dataMall...

Re: How MTA shut down my app for Penn Station commuters

#28

Post by vinnycoyne »

Sir_Cmpwn wrote: post #22
karrotwaltz wrote: post #11

Earlier quoted context omitted.

> They also stopped publishing the track number on the website — so going back to scraping was not an option either. So the only way to gather the data would be manually from the station.

Carefully placed raspberry pi + OCR?

I'd imagine that placing a "homemade" electronic device in a densely-populated, public space is going to attract a lot of unwanted attention.

Re: How MTA shut down my app for Penn Station commuters

#29

Post by tedmiston »

sologoub wrote: post #16
midnitewarrior wrote: post #10

Why didn't he switch to a crowd-sourced data model like GasBuddy?

Was thinking the exact same thing. One possible reason would be wrong info, but you could use some sort of weighted algorithm that would need more than one submission to match and/or trust level per submitter based on historical correctness.

Exactly. Like what Yelp / Foursquare / Google Places use.

Re: How MTA shut down my app for Penn Station commuters

#30

Post by Spooky23 »

state_machine wrote: post #4

I can see why the MTA wouldn't want you telling people to go to a track before it is actually assigned -- if the train ends up on a different track, now you need to get all those people back up, off that platform an on to the new one, clogging stairs that that people who actually wanted that platform might be trying to use too. Obviously it would be nice to assign tracks earlier, so that people can head straight to t…

I don't use the LIRR tracks often, but many of the Amtrak trains arrive at the same track or 1-2 tracks. Veteran travelers know where to go, casual travelers are stuck in this maze that is very difficult to navigate.
Post reply on HN