Live data from Hacker News

Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

engineering.atspotify.com

101–110 of 135 posts

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#101

“How did ” “Enjoy work like this? Want to make a big impact as a lowly programmer who will never start their own company? Well at , lowly programmers who will never start their own companies have the freedom to make a big impact.”

Spotify is no longer a startup.

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#102
post #73

And they’re still less useful than the data last.fm makes available to you.

Useful how? What actionable insights do you get from Last.fm? It's vanity metrics optimized for sharing on social networks and there's nothing wrong with it.

Well, for one it gives me the yearly summary once the year is actually over.

It's not particularly actionable to know you listen to more music on a Saturday than a Sunday, but it is mildly interesting for those who are curious about such things.

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#103
post #88
post #79

Earlier quoted context omitted.

Surely you read the article before posting, right? From literally the first sentence: > from our largest Dataflow job

Surely you have read the guidlines before posting, right? > Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that." https://news.ycombinator.com/newsguidelines.html The hackernews title and the article title say "the". Critizing this clickbait is more than warranted.

Fair point, I could have worded it better.

But I think it's a reach to call this "clickbait".

Article titles are shortened all the time, and you can't expect them to have all the context in the title.

However, one should reasonably expect people participating in a discussion about an article (particularly when posting criticism) to have actually read it.

Complaining about something that is provably false in the first sentence of the article is the bigger sin here, is it not?

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#104
Perhaps a bit off-topic. But a lot of users (myself included) reported wildly inaccurate data in the spotify wrapped this year with seemingly no explanation, no shared accounts, no re-used passwords, no weird listening history, etc.

I wonder if some of the data in the "We worked with the maintainer of these data sets to convert a year’s worth of data to SMB format." step got corrupted or just wrongly converted/lost.

I'm not sure how else explain that I have to google artists in my top 10 because I never heard of them.

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#105

I came here expecting to read about the tech in the article or how others do big data processing stuff. Instead I get off topic Spotify rants.. Did you read the article or just see Spotify in the headline and decided your gripes therefore are relevant?

The article isn't easy to read unless you have some knowledge up front about the used technologies and what the problem is.

This definitely drives people to comment on other things.

My gut feeling screams they made a problem themselves in the first place which they then "solved". Similar to a "solution running around looking for a problem" type of deal.

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#106
post #57

Earlier quoted context omitted.

Really? I find my 'Discover Weekly' playlist to be astounding. I have to confess for a long time I thought it was a human curated playlist from someone who just happened to have exactly the sames tastes as me.

I have to second this. I can't list the number of artists I have discovered over the last few years using Discover Weekly alone. It's really incredible.

It's been the opposite for me. It regularly surfaces songs I've listened to in the past from artists I listen to frequently. As a music discovery algo Spotify has been nothing but poor in my opinion.

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#108

Largest data flow job ever? I’m sure Google would beg to differ. At Quantcast we process 50PB every day, and that’s nothing compared to real scale like Google. And merge joins from sorted data? Joins have been done that way since the punched card days on mainframes (and by any scaled data system)

Misleading headline, the first sentence is "how Spotify optimized and sped up elements from *our largest Dataflow job*". Surely it's not the largest ever run, even on Dataflow.

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#109

Earlier quoted context omitted.

This has become a huge problem on HN lately. Lots of discussions are nothing but complaining. Now the technical discussions are starting to get infested with off-topic whining. The mods don't do anything about off-topic rants. If you point it out you'll get downvoted [1][2][3]. [1] https://news.ycombinator.com/item?id=25839399 [2] https://news.ycombinator.com/item?id=25064636 [3] https://news.ycombinator.com/item?id=…

> The mods don't do anything about off-topic rants. Mod. There is a question of how much one moderator can do against the tide. HN really needs a couple of full time paid moderators, with their salaries covered by the zillion dollar YC bank account.

[deleted]

Re: Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

#110
post #37
post #32

Earlier quoted context omitted.

I wonder if it'll meet the same fate as Netflix at some point, with publishing houses going for their own streaming services instead. I don't think so because the way that people consume music and the way they consume films and television are very different. With a film you might block out a few hours to watch that specific provider. With music you're more likely to want to interleave content from several providers a…

A considerable amount of music is distributed by a small subset of providers. So in that regard it's not that different to the TV / film situation and you could theoretically still interleave different artists. There's also a common use case where people will just play a specific artist for an hour. Or even an album. Frankly, I hope services like Spotify don't disappear. It's a great loss to consumers just how fragme…

I don't think that behavior is all to common. I see people listening to a very wide range of artists and almost never reach for stuff outside of it around here.
Post reply on HN