Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

engineering.atspotify.com

Post reply on HN
Print view View original

91–100 of 135 posts

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#91

Post by yaqoob »

krakmh wrote: post #90
shaicoleman wrote: post #70

My off-topic rant: I'd really wish Spotify would focus on improving the core player experience. It has barely seen any improvements in years. * Not overwrite/delete my listening history everytime I switch devices * Allow tabs, or some way to resume what I've been listening to in different contexts * Option to open only one instance, instead of having multiple instances that mess with each other * Playing local files…

You are absolutely right. Even more, people are using it for free using mods like these https://bestforandroid.com/apk/spotify-premium-mod-apk/

Yup, well said.

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#93

Post by davweb »

paulsutter wrote: post #75

Largest data flow job ever? I’m sure Google would beg to differ. At Quantcast we process 50PB every day, and that’s nothing compared to real scale like Google. And merge joins from sorted data? Joins have been done that way since the punched card days on mainframes (and by any scaled data system)

It's largest Dataflow[1] job ever, with a capital D, not "data flow".

[1]: https://cloud.google.com/dataflow

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#94

Post by adventured »

johncena33 wrote: post #74
matsemann wrote: post #30

I came here expecting to read about the tech in the article or how others do big data processing stuff. Instead I get off topic Spotify rants.. Did you read the article or just see Spotify in the headline and decided your gripes therefore are relevant?

This has become a huge problem on HN lately. Lots of discussions are nothing but complaining. Now the technical discussions are starting to get infested with off-topic whining. The mods don't do anything about off-topic rants. If you point it out you'll get downvoted [1][2][3]. [1] https://news.ycombinator.com/item?id=25839399 [2] https://news.ycombinator.com/item?id=25064636 [3] https://news.ycombinator.com/item?id=…

> The mods don't do anything about off-topic rants.

Mod. There is a question of how much one moderator can do against the tide. HN really needs a couple of full time paid moderators, with their salaries covered by the zillion dollar YC bank account.

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#95

Post by npteljes »

spotyoufi462881 wrote: post #21

Anyone wanna know how much Spotify wanna know about you? https://twitter.com/steipete/status/1025024813889478656

I'm already thinking that every service hoovers as much up as they can. Nice to see proof that it is actually happening!

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#96

Post by npteljes »

josteink wrote: post #31
fooblat wrote: post #8

When this report came out it was the straw that broke the camel's back for me in terms of my data privacy. Most people seem to have found Wrapped 2020 entertaining but I found it creepy. I miss being able to do something simple like listen to music or watch a movie without all my actions being recorded and saved. So I'm back to buying physical media and DRM free downloads. I'm convinced that it is now important to ho…

> I'm convinced that it is now important to hold on to older appliances that work without internet access or data collection Or run modern, up to date FOSS equivalents on machines you control. I’ve migrated more and more services like that and I’m slowly but surely building my own “cloud”.

I understand why people don't constantly think about their digital footprint, but it's not that different from when they order the same thing at a stall for a month, and the clerk begins to ask "Do you want your usual?".

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#97

Post by enigmo »

BorisTheBrave wrote: post #65

I'm having a hard time understanding this article. It seems to be a bit too low level on the specifics of Beam for general consumption. From what i undestand, Spark has the same feature built in. If the planner knows that the source data is partitioned and/or sorted appropriately, it can skip shuffling/sorting it, instead having each executor directly requesting the one file it needs. It's a nice optimization, but it…

Hive also has had this optimization for as long as I can remember. As others have noted it's not particularly new or novel, it's just not part of the Beam SDK.

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#98

Post by matsemann »

chishaku wrote: post #53
matsemann wrote: post #30

I came here expecting to read about the tech in the article or how others do big data processing stuff. Instead I get off topic Spotify rants.. Did you read the article or just see Spotify in the headline and decided your gripes therefore are relevant?

You discover that people have different thoughts than you and decided your gripe about this is therefore relevant? Upvote, downvote or move on.

I agree that my post is just additional noise. But it's noise not disturbing a good signal.

Let me just point out that I think discussing sides of what's linked can be interesting and relevant. In this instance I think discussing privacy around the amount of data Spotify stores is a relevant subdiscussion worth exploring. But complaints about their UI doesn't feel very relevant. Etc.

Now that's only my opinion. What made me write my comment was that it was literally no one discussing the concepts of the article in the first 15 or so comments. Which I found a bit disappointing as I thought the tech is interesting.

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#99

Post by harryf »

mandis wrote: post #68
harryf wrote: post #62

Earlier quoted context omitted.

Agreed to I'll bite... > The intuition is that for datasets commonly and frequently joined on a known key, e.g., user events with user metadata on a user ID, we can write them in bucket files with records bucketed and sorted by that key. By knowing which files contain a subset of keys and in what order, shuffle becomes a matter of merge-sorting values from matching bucket files, completely eliminating costly disk and…

>It reminds me of something in Eric Raymonds "The Art of Unix Programming" (I don't have time to find the link right now) where it discussed an approach from the earlier days of Linux filesystems where you had a limit on the number of iNodes that could exist in a single directory and corresponding performance. The work around was to create a subdirectory structure to store files based on the filename. But then you te…

I think the bit I was remembering was the Terminfo Case Study on page 149 of the Art of Unix Programming - https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.62... - read it a long time ago though and re-reading now, it's not exactly what I was remembering but that's memory for you...
Post reply on HN