Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

engineering.atspotify.com

Post reply on HN
Print view View original

71–80 of 135 posts

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#71

Post by max_streese »

jsjsbdkj wrote: post #66
max_streese wrote: post #2

Hi not sure if I am just completely off here but I am wondering how this relates or compares to processing things with Kafka and Kafka Streams? If I am reading things correctly with Kafka the workflow equivalent to what's written in the article would be to have your producer produce via hash-based-round-robin (the default partitioning algorithm) based on the key you are interested in into some topic and then your con…

This is the most basic pattern for distributed joins - you hash on the join key in both tables and shuffle data based on hash ranges. In some systems like Redshift you can designate the key for distribution so that "related" records are already co-located on a single shard. > our data would already be sorted for the given keys (because within a partition Kafka has sorting guarantees) It's been a while since I used Ka…

Yes I guess my point is when using Kafka in combination with Kafka Streams and you produce things partitioned in a way that you need them for consumption then you do not need to do any shuffling in the instance where you want to join because data is already partitioned correctly.

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#72

Post by fooblat »

diggan wrote: post #35
fooblat wrote: post #8

When this report came out it was the straw that broke the camel's back for me in terms of my data privacy. Most people seem to have found Wrapped 2020 entertaining but I found it creepy. I miss being able to do something simple like listen to music or watch a movie without all my actions being recorded and saved. So I'm back to buying physical media and DRM free downloads. I'm convinced that it is now important to ho…

Seems strange that "Wrapped 2020" was the straw for you. Ever since launch, the top reasons for starting to use Spotify has been availability (across devices) and the provided radio/discover playlists that automatically finds music based on your previous history, this has always been a core proposition of Spotify, and not something that happened now. Then it's not always good, right or even close sometimes, but it's…

I joined spotify in 2011 and the core offer was simply access to all music across your devices. However, that's not really relevant.

2020 was the year I really started to question why I was taking such care with my data in some ways but not others. The Wrapped 2020 was a bright reminder to me that if I want to take my privacy seriously I need to look at everything in my life that collects data. Simple as that.

I don't think spotify is wrong or evil to collect the data. I actually think it is a great product. I have just decided that I want to leave as little data around about my daily activities as possible.

As it happens I don't use the Discovery features very much and it turns out there are still some enjoyable FM stations where I live. When I want background music I turn on the radio. When I want something more specific I play an album or playlist from my collection.

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#73

Post by dewey »

iamacyborg wrote: post #5

And they’re still less useful than the data last.fm makes available to you.

Useful how? What actionable insights do you get from Last.fm? It's vanity metrics optimized for sharing on social networks and there's nothing wrong with it.

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#74

Post by johncena33 »

matsemann wrote: post #30

I came here expecting to read about the tech in the article or how others do big data processing stuff. Instead I get off topic Spotify rants.. Did you read the article or just see Spotify in the headline and decided your gripes therefore are relevant?

This has become a huge problem on HN lately. Lots of discussions are nothing but complaining. Now the technical discussions are starting to get infested with off-topic whining. The mods don't do anything about off-topic rants. If you point it out you'll get downvoted [1][2][3].

[1] https://news.ycombinator.com/item?id=25839399 [2] https://news.ycombinator.com/item?id=25064636 [3] https://news.ycombinator.com/item?id=24699908

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#75

Post by paulsutter »

Largest data flow job ever? I’m sure Google would beg to differ. At Quantcast we process 50PB every day, and that’s nothing compared to real scale like Google.

And merge joins from sorted data? Joins have been done that way since the punched card days on mainframes (and by any scaled data system)

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#76

Post by texasbigdata »

BorisTheBrave wrote: post #65

I'm having a hard time understanding this article. It seems to be a bit too low level on the specifics of Beam for general consumption. From what i undestand, Spark has the same feature built in. If the planner knows that the source data is partitioned and/or sorted appropriately, it can skip shuffling/sorting it, instead having each executor directly requesting the one file it needs. It's a nice optimization, but it…

50% cost reduction though

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#77

Post by spotyoufi462881 »

Jonnax wrote: post #61
spotyoufi462881 wrote: post #21

Anyone wanna know how much Spotify wanna know about you? https://twitter.com/steipete/status/1025024813889478656

What's scary about this? They're complying with GDPR. Isn't that a good thing? The "scary" thing in that tweet is that they store the manufacturer of their bluetooth headphones?

That's not what one could call complying. He had to follow them like a dog for a long while just to get his rightful data.

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#79

Post by sbarre »

paulsutter wrote: post #75

Largest data flow job ever? I’m sure Google would beg to differ. At Quantcast we process 50PB every day, and that’s nothing compared to real scale like Google. And merge joins from sorted data? Joins have been done that way since the punched card days on mainframes (and by any scaled data system)

Surely you read the article before posting, right?

From literally the first sentence:

> from our largest Dataflow job

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#80

Post by sbarre »

polskibus wrote: post #18

I wonder if they could publish dollar cost of that job before and after the optimization, as provided by GCP billing. I know it could be a bit unfair (some costs may be static, regardless of job size, etc.) but it would improve decision making for others if discussions of public cloud usage optimizations also include the cost.

They do refer to savings in percentages.. I feel like giving away actual dollar costs would potentially break contractual agreements (because I doubt Spotify pays public list price), and potentially give away competitive information about how much data they have on their customers etc..

I agree with you that it would be interesting to know, I just don't think it's realistic for them to release that information.

Post reply on HN