Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fun with NFL Stats, Bokeh, and Pandas

j253.github.io

Post reply on HN
Print view View original

11–20 of 28 posts

Re: Fun with NFL Stats, Bokeh, and Pandas

#11

Post by J253 »

sndean wrote: post #10

> The small spikes at 5 yard increments is interesting and I don't really have a good explanation other than to think that whoever recorded the yardage data liked rounding to the nearest 5 if it was close. Anyone else have any other ideas? I'll go with the idea that the refs are biased with their ball placement and tend to put the ball on lines [0]. Also, players/teams practice, speak, think in 5 yard increments, so…

Thanks for the link! That's a great resource and write up about the statistical heaping effect. I'll include that in a post update.

Re: Fun with NFL Stats, Bokeh, and Pandas

#12

Post by arglebarnacle »

Does anyone have any insight about what kind of jobs are out there for people with the kind of skills demonstrated in this post?

I have a lot of data exploring, cleaning and visualizing skills, python/SQL skills and experience using it to make business decisions, but this type of thing falls short of what most people would consider "data science"

Re: Fun with NFL Stats, Bokeh, and Pandas

#13

Post by diminoten »

burntsushi wrote: post #9
nubb wrote: post #6

I've always enjoyed this project for pulling nfl stats. https://github.com/BurntSushi/nflgame

That project is no longer maintained because I don't use it any more, but others have picked up the baton: https://github.com/derek-adair/nflgame Back in the day, I used nflgame along with https://github.com/BurntSushi/nfldb https://github.com/BurntSushi/nflvid https://github.com/BurntSushi/nflfan to setup a simple local web UI that allowed me to quickly search through every play and watch any single play I wanted .…

We briefly spoke via GitHub about a month ago, and during that convo (it was in a ticket), you mentioned that the source has inaccuracies. Is there any elaboration there or do the NFL people use a different data source to do things like Fantasy and official stats?

Re: Fun with NFL Stats, Bokeh, and Pandas

#14

Post by dyim »

sndean wrote: post #10

> The small spikes at 5 yard increments is interesting and I don't really have a good explanation other than to think that whoever recorded the yardage data liked rounding to the nearest 5 if it was close. Anyone else have any other ideas? I'll go with the idea that the refs are biased with their ball placement and tend to put the ball on lines [0]. Also, players/teams practice, speak, think in 5 yard increments, so…

ha! that's funny. "We've got to get to the 35 yard line, so Morten can kick a 53-yarder, so we can have a chance."

Re: Fun with NFL Stats, Bokeh, and Pandas

#15

Post by J253 »

arglebarnacle wrote: post #12

Does anyone have any insight about what kind of jobs are out there for people with the kind of skills demonstrated in this post? I have a lot of data exploring, cleaning and visualizing skills, python/SQL skills and experience using it to make business decisions, but this type of thing falls short of what most people would consider "data science"

Agreed. And I definitely make no claims about this being earth-shattering "data science". I just happened spend a few hours over the weekend making some plots and commenting about what I saw with some Python tools.

I'll also state that I am neither a data scientist nor a statistician. I'm a Python application engineer with a background in mechanical engineering, so that might help set the context a bit more.

Re: Fun with NFL Stats, Bokeh, and Pandas

#16

Post by eunoia »

burntsushi wrote: post #9
nubb wrote: post #6

I've always enjoyed this project for pulling nfl stats. https://github.com/BurntSushi/nflgame

That project is no longer maintained because I don't use it any more, but others have picked up the baton: https://github.com/derek-adair/nflgame Back in the day, I used nflgame along with https://github.com/BurntSushi/nfldb https://github.com/BurntSushi/nflvid https://github.com/BurntSushi/nflfan to setup a simple local web UI that allowed me to quickly search through every play and watch any single play I wanted .…

I learned Postgres while playing around with nfldb years ago. Great experience.

Thank you for all your work, those are some very impressive projects.

Re: Fun with NFL Stats, Bokeh, and Pandas

#17

Post by burntsushi »

diminoten wrote: post #13
burntsushi wrote: post #9

Earlier quoted context omitted.

That project is no longer maintained because I don't use it any more, but others have picked up the baton: https://github.com/derek-adair/nflgame Back in the day, I used nflgame along with https://github.com/BurntSushi/nfldb https://github.com/BurntSushi/nflvid https://github.com/BurntSushi/nflfan to setup a simple local web UI that allowed me to quickly search through every play and watch any single play I wanted .…

We briefly spoke via GitHub about a month ago, and during that convo (it was in a ticket), you mentioned that the source has inaccuracies. Is there any elaboration there or do the NFL people use a different data source to do things like Fantasy and official stats?

I'm not an NFL insider. I don't know what they do internally. I only know that 1) the undocumented NFL GameCenter JSON is not 100% accurate and that 2) any user of a fantasy league would notice these inaccuracies. I did a test a while back by comparing GameCenter data with Yahoo's data. Kickers tend to have the most inaccuracies: https://github.com/BurntSushi/nflgame/blob/master/test-data/... QB stats are more solid for example, but there are still minor problems: https://github.com/BurntSushi/nflgame/blob/master/test-data/...

From those observations, you can't really make any solid conclusions. But if you think about it for a bit, you might be able to reason your way to some guesses. For example, one possibility is that the GameCenter data is NFL's own construction that's only used for their GameCenter interfaces, where as places that "official" data is needed might be powered by Elias[1]. Why the discrepancy? Again, I don't know. It could be legacy software related. It could be contract/legal related. Or it could jus tbe plain old bugs. e.g., Maybe GameCenter hooks into an initial lossy but fast feed that is updated during the game, but never receives updates from a slower but more accurate feed later.

Or maybe the NFL purposely inserts data canaries because they know this JSON feed is unprotected, and they intend on using those data canaries to detect folks using their data in an unlicensed fashion. I'm pretty sure IMDb does this, for example. Or maybe they just insert errors purposely to make it too costly for anyone to use this data in situations that require 100% accuracy (like fantasy football leagues).

My guess is some innocuous blend of legal and legacy software reasons.

[1] - http://www.esb.com/

Re: Fun with NFL Stats, Bokeh, and Pandas

#19

Post by kilbuz »

arglebarnacle wrote: post #12

Does anyone have any insight about what kind of jobs are out there for people with the kind of skills demonstrated in this post? I have a lot of data exploring, cleaning and visualizing skills, python/SQL skills and experience using it to make business decisions, but this type of thing falls short of what most people would consider "data science"

In my experience, certainly a large number of jobs advertised at 'data scientist' would be exactly as you describe. Emphasis on the 'cleaning' part.

Re: Fun with NFL Stats, Bokeh, and Pandas

#20

Post by dataanalyst1 »

arglebarnacle wrote: post #12

Does anyone have any insight about what kind of jobs are out there for people with the kind of skills demonstrated in this post? I have a lot of data exploring, cleaning and visualizing skills, python/SQL skills and experience using it to make business decisions, but this type of thing falls short of what most people would consider "data science"

Data Analyst
Post reply on HN