Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bad Data and Data Engineering: Dissecting Google Play Music Takeout Data

chollinger.com

Post reply on HN
Print view View original

11–16 of 16 posts

Re: Bad Data and Data Engineering: Dissecting Google Play Music Takeout Data

#11

Post by randomifcpfan »

progbits wrote: post #2

So are the mp3 files not the same as what the author uploaded? I could imagine weird organization for tracks from the service but for self-uploaded data I would be surprised if they didn't just give them back the same. The article never mentioned how this showed up in the GPM app itself which feels lacking. Otherwise a nice article but it reminds me why I long ago gave up on media metadata organization. So much work,…

IIRC, GPM stored user uploads in MP3. If you uploaded a non-MP3 file, it was transcoded into a MP3 during upload. It is this file that GPM takeout is providing.

Separate from that, GPM matched your uploaded MP3 file against the service music corpus, and if there was a match, the service streamed the canonical version. Originally the streaming service used 320 kbps MP3, but later the service switched to 256 kbps AAC. GPM takeout does not provide the canonical version.

Re: Bad Data and Data Engineering: Dissecting Google Play Music Takeout Data

#12

Post by faizshah »

diarrhea wrote: post #7
faizshah wrote: post #5

Earlier quoted context omitted.

The makefile data pipeline is definitely an underrated technique a couple great HN comments on this technique: - https://news.ycombinator.com/item?id=22283368 - https://news.ycombinator.com/item?id=18896204 I personally learned it from bioinformaticians theres great coverage of this and other command line data skills in this book: https://www.oreilly.com/library/view/bioinformatics-data-ski... The SQLite, pandas, bas…

Great read on Jupyter Notebooks. I've always had a strong dislike for them as well - just feel fuzzy, dirty, often in weird state - and the user put it into words nicely. I might just be bad with Jupyter, but I've been willing to mold these notebooks into saner forms (modularization, keeping state, deterministic cell order/content/execution) and came up empty. At some point you got to ask if it's not the tool's fault…

You are absolutely right I have taken to referring to that as “notebook code.”

I love jupyter because its a great interactive programming environment that speeds up building scripts. But jupyter notebooks always end up being…notebooks. They are records of past work and thoughts that I had but I wouldn't present my paper notes to a coworker as finished documentation. I always end up extracting those jupyter snippets into runnable scripts and then I end up doing a lot of the work all over again as I parameterize things.

I have taken to using pycharm scientific mode and adding “cells” into my scripts: https://blog.jetbrains.com/pycharm/2018/04/pycharm-scientifi...

You get the interactivity of jupyter with the cells, the scientific view for plots and data, but you’re also writing a real script so you can still maintain some sanity in your code and good git history (and hopefully add some tests).

Then I combine these scripts together into a makefile and voila quick, easy and maintainable scripts. If you just stick to make, python, pandas and bash any programmer can modify your pipeline without needing specialist skills. I have written scrapers + ETL in God I wish I still worked in data science I’m currently a front end dev and don’t get to do any of this at work. It’s such fun and offers so much room for creativity and problem solving.

Re: Bad Data and Data Engineering: Dissecting Google Play Music Takeout Data

#13

Post by wodenokoto »

faizshah wrote: post #12
diarrhea wrote: post #7

Earlier quoted context omitted.

Great read on Jupyter Notebooks. I've always had a strong dislike for them as well - just feel fuzzy, dirty, often in weird state - and the user put it into words nicely. I might just be bad with Jupyter, but I've been willing to mold these notebooks into saner forms (modularization, keeping state, deterministic cell order/content/execution) and came up empty. At some point you got to ask if it's not the tool's fault…

You are absolutely right I have taken to referring to that as “notebook code.” I love jupyter because its a great interactive programming environment that speeds up building scripts. But jupyter notebooks always end up being…notebooks. They are records of past work and thoughts that I had but I wouldn't present my paper notes to a coworker as finished documentation. I always end up extracting those jupyter snippets i…

I think notebooks are great for building a presentation / reports of a data exploration, but I agree that they shouldn't represent programs.

Rstudio won't export an R-markdown notebook to PDF without rerunning all cells sequentially. It's great for keeping the code "in check", but for things that takes hours to run, it can be quite annoying.

I do think most of the problems with notebooks go away if you somehow force sequential running before presenting it to other people.

Another really cool take on notebooks are Julia's Pluto Notebooks. Like a spreadsheet, there is no sequence that cells are run in. Everything is updated simultaneously. It's kinda hard to explain, but the JuliacCn presentation on these is absolutely wonderful: https://www.youtube.com/watch?v=IAF8DjrQSSk

Re: Bad Data and Data Engineering: Dissecting Google Play Music Takeout Data

#14

Post by otter-in-a-suit »

wodenokoto wrote: post #4
faizshah wrote: post #3

Great post, for this pipeline I would have probably used a makefile for the batch pipeline instead of airflow just to keep it simple. I would also make my sink a SQLite database so that you can easily search through it with a web interface using datasette. For the places where bash was used I would just use python and any cli tools you want to call I just use subprocess. It’s much simpler and I can run the scripts in…

My first thought was also "why not SQLite?", but the author says he already has a MariaDB running. So, using the tools you know. I guess it is the same for make vs airflow. I had no idea they could be used interchangeably for single machine workloads. While I've seen datasette mentioned a lot of places, I still don't really know what it is, but if it makes exploring sqlite databases easy, I should give it a try!

I use SQLite quite a bit and I think it's fantastic, especially since Hipp seems to be a wonderful guy (judging by The Changelog podcast).

And you're right - use the tools you know and have running. I have all sorts of schemas and tables on that old instance, since I tend to use it if I need "anything SQL" - when I'm at home, is it as easy as using SQLite. My latest article used Trino and Hadoop-adjacent stuff... while fascinating it its own right, sometimes it's nice to just say "jdcb:// ..." :)

Re: Bad Data and Data Engineering: Dissecting Google Play Music Takeout Data

#15

Post by otter-in-a-suit »

faizshah wrote: post #5
wodenokoto wrote: post #4

Earlier quoted context omitted.

My first thought was also "why not SQLite?", but the author says he already has a MariaDB running. So, using the tools you know. I guess it is the same for make vs airflow. I had no idea they could be used interchangeably for single machine workloads. While I've seen datasette mentioned a lot of places, I still don't really know what it is, but if it makes exploring sqlite databases easy, I should give it a try!

The makefile data pipeline is definitely an underrated technique a couple great HN comments on this technique: - https://news.ycombinator.com/item?id=22283368 - https://news.ycombinator.com/item?id=18896204 I personally learned it from bioinformaticians theres great coverage of this and other command line data skills in this book: https://www.oreilly.com/library/view/bioinformatics-data-ski... The SQLite, pandas, bas…

makefiles in general are something that I quite frankly never figured people use with anything (forgive me for saying it) but "old-school C"... but I started using them earlier this year (the linked article is from Feb) after finding a makefile in a Python GitHub repo. I've since created one for most of my Python and go repositories and they've been a great tool. It has, however, not occurred to me to use them in place of actual pipelines (as the linked comments suggest), so I'll definitely try this soon, thanks!

That being said, a lot of what I do on my blog is stuff I don't get to do at work - Airflow being one of those things that has been "abstracted away from me" for too long...

Re: Bad Data and Data Engineering: Dissecting Google Play Music Takeout Data

#16

Post by otter-in-a-suit »

wodenokoto wrote: post #9

> The script should be decently self-explanatory [...] Please note that this is all single-threaded, which I don’t recommend - with nohup and the like, you can trivially parallelize this. How do you parallelize a loop in bash without getting all the echo's intertwined and jumbled together?

Maybe not nohup itself, but I often do simple background tasks + a blocking `wait` statement in a loop, for instance here: https://github.com/chollinger93/debian-scripts/blob/master/u...

If you don't use nohup, you can just pipe stdout and err to a different file descriptor for logs within said loop.

Or, of course, script something in a language of your choice.

Post reply on HN