61–70 of 135 posts
Anyone wanna know how much Spotify wanna know about you? https://twitter.com/steipete/status/1025024813889478656
They're complying with GDPR. Isn't that a good thing?
The "scary" thing in that tweet is that they store the manufacturer of their bluetooth headphones?
I came here expecting to read about the tech in the article or how others do big data processing stuff. Instead I get off topic Spotify rants.. Did you read the article or just see Spotify in the headline and decided your gripes therefore are relevant?
> The intuition is that for datasets commonly and frequently joined on a known key, e.g., user events with user metadata on a user ID, we can write them in bucket files with records bucketed and sorted by that key. By knowing which files contain a subset of keys and in what order, shuffle becomes a matter of merge-sorting values from matching bucket files, completely eliminating costly disk and network I/O of moving key–value pairs around.
I'm actually surprised that this should be regarded as "novel" in data science.
It reminds me of something in Eric Raymonds "The Art of Unix Programming" (I don't have time to find the link right now) where it discussed an approach from the earlier days of Linux filesystems where you had a limit on the number of iNodes that could exist in a single directory and corresponding performance. The work around was to create a subdirectory structure to store files based on the filename. But then you tended to get many files starting with the same characters all in the same directories. What turned out to be a better way to distribute the files evenly in the directory structure was to take the first and _last_ character of the file name and use those to create the subdirectories. This way you were more likely to spread the files evenly across the structure.
Wrapped works so well because it panders to you. Everyone likes to be acknowledged for listening to the weird indie band they discovered earlier this year. I enjoy it anyways, and Spotify is still a great service for now - I wonder if it'll meet the same fate as Netflix at some point, with publishing houses going for their own streaming services instead.
I wonder if it'll meet the same fate as Netflix at some point, with publishing houses going for their own streaming services instead. I don't think so because the way that people consume music and the way they consume films and television are very different. With a film you might block out a few hours to watch that specific provider. With music you're more likely to want to interleave content from several providers a…
Spotify is a company that feels like they want to be a "big tech company" when in reality they do not need to. All they need to do is provide a great service with as much music as possible.
I for one love discover weekly and hope they will continue working on improving it. I consider it a core feature :)
From what i undestand, Spark has the same feature built in. If the planner knows that the source data is partitioned and/or sorted appropriately, it can skip shuffling/sorting it, instead having each executor directly requesting the one file it needs.
It's a nice optimization, but it's not game changing. You often end up having to shuffle anyway, as you are joining on a different key, or for performance reason you need more executors than the set amount of partitions, or the shuffle needed to write the data doesn't justify the savings on the readers.
Maybe it's better with their additional optimizations? Spark does not do those, mostly.
Hi not sure if I am just completely off here but I am wondering how this relates or compares to processing things with Kafka and Kafka Streams? If I am reading things correctly with Kafka the workflow equivalent to what's written in the article would be to have your producer produce via hash-based-round-robin (the default partitioning algorithm) based on the key you are interested in into some topic and then your con…
> our data would already be sorted for the given keys (because within a partition Kafka has sorting guarantees)
It's been a while since I used Kafka but I don't remember "sorting guarantees". Consumers see events "in order" based on when they were produced, because each partition is a queue.
Earlier quoted context omitted.
Speaking of interleaving, I wish Spotify understood how to interact with Concept Albums. Changing to the middle of The Wall for one song is jarring and I usually skip it. And then I worry skipping it is going to train the algorithm that I don't like Pink Floyd.
What does this mean? I don't use spotify. Are you saying when you go to listen to an album it will put a random song into the queue?
They probably meant listening in random mode, and Spotify randomly choosing a track that doesn’t make much sense outside the context of its album.
I came here expecting to read about the tech in the article or how others do big data processing stuff. Instead I get off topic Spotify rants.. Did you read the article or just see Spotify in the headline and decided your gripes therefore are relevant?
Agreed to I'll bite... > The intuition is that for datasets commonly and frequently joined on a known key, e.g., user events with user metadata on a user ID, we can write them in bucket files with records bucketed and sorted by that key. By knowing which files contain a subset of keys and in what order, shuffle becomes a matter of merge-sorting values from matching bucket files, completely eliminating costly disk and…
Interesting. I have been pondering over filesystem performance and inode limits in servers/home-servers since a long time. This seems useful infomration
* Not overwrite/delete my listening history everytime I switch devices
* Allow tabs, or some way to resume what I've been listening to in different contexts
* Option to open only one instance, instead of having multiple instances that mess with each other
* Playing local files crashes/not working on Linux
* Change playback speed, not just for podcasts
* Jump back/forward, not just for podcasts
* Have some visibility when the song was last played / play count
* Liked songs not always appearing in search results
* Sorting search results not working
* Add basic functionality to the dbus interface (e.g. seeking)
* Ability to report songs (e.g. wrong titles/badly split tracks/etc.)