121–130 of 135 posts
“How did ” “Enjoy work like this? Want to make a big impact as a lowly programmer who will never start their own company? Well at , lowly programmers who will never start their own companies have the freedom to make a big impact.”
Spotify is no longer a startup.
I came here expecting to read about the tech in the article or how others do big data processing stuff. Instead I get off topic Spotify rants.. Did you read the article or just see Spotify in the headline and decided your gripes therefore are relevant?
Agreed to I'll bite... > The intuition is that for datasets commonly and frequently joined on a known key, e.g., user events with user metadata on a user ID, we can write them in bucket files with records bucketed and sorted by that key. By knowing which files contain a subset of keys and in what order, shuffle becomes a matter of merge-sorting values from matching bucket files, completely eliminating costly disk and…
Index organized tables in Oracle, clustered tables in Mssql. "Intuition" in modern big data world :)
I came here expecting to read about the tech in the article or how others do big data processing stuff. Instead I get off topic Spotify rants.. Did you read the article or just see Spotify in the headline and decided your gripes therefore are relevant?
Agreed to I'll bite... > The intuition is that for datasets commonly and frequently joined on a known key, e.g., user events with user metadata on a user ID, we can write them in bucket files with records bucketed and sorted by that key. By knowing which files contain a subset of keys and in what order, shuffle becomes a matter of merge-sorting values from matching bucket files, completely eliminating costly disk and…
Earlier quoted context omitted.
You discover that people have different thoughts than you and decided your gripe about this is therefore relevant? Upvote, downvote or move on.
I agree that my post is just additional noise. But it's noise not disturbing a good signal. Let me just point out that I think discussing sides of what's linked can be interesting and relevant. In this instance I think discussing privacy around the amount of data Spotify stores is a relevant subdiscussion worth exploring. But complaints about their UI doesn't feel very relevant. Etc. Now that's only my opinion. What…
The signal will always be weak early on.
Look at the comments now and it's clear that the upvote/downvote mechanisms are working sufficiently to address your concern.
This should not be surprising. It's typically faster to not read the article and spew superficial off-topic comments than to write on-topic, substantive, and technical comments.
Earlier quoted context omitted.
This has become a huge problem on HN lately. Lots of discussions are nothing but complaining. Now the technical discussions are starting to get infested with off-topic whining. The mods don't do anything about off-topic rants. If you point it out you'll get downvoted [1][2][3]. [1] https://news.ycombinator.com/item?id=25839399 [2] https://news.ycombinator.com/item?id=25064636 [3] https://news.ycombinator.com/item?id=…
Absolutely, there has been a dramatic shift in the type of people who visit HN in the past years. I used to think that Reddit was bad in this regard but to be honest it mostly affects the big subreddits, the niche and small ones still have a high quality community. HN became pretty much like the biggest subs on Reddit.
This comment thread is in its own category of low quality discussion.
Negativity bias prevents you from seeing that 95% of the homepage right now is technical/nerdy with a lot of high quality corresponding discussion.
When political/social issues hit the homepage, they often slide off quickly if the corresponding discussion is of low quality (has many downvoted comments).
HN is certainly not perfect but just focusing on the parts you don't like prevents you from seeing the bigger picture.
Earlier quoted context omitted.
A considerable amount of music is distributed by a small subset of providers. So in that regard it's not that different to the TV / film situation and you could theoretically still interleave different artists. There's also a common use case where people will just play a specific artist for an hour. Or even an album. Frankly, I hope services like Spotify don't disappear. It's a great loss to consumers just how fragme…
I don't think that behavior is all to common. I see people listening to a very wide range of artists and almost never reach for stuff outside of it around here.
My wife uses them to shuffle singles by specific artists (she’s more into pop music).
I’d wager if my wife and I both coincidentally follow the same pattern despite doing so for different reasons, that it’s then likely a more common pattern than first assumed. Please also bare in mind that I’m not suggesting our use cases are how the majority of people consume music, but I’d be surprised if it was small enough to be a rounding error.
Earlier quoted context omitted.
I agree that my post is just additional noise. But it's noise not disturbing a good signal. Let me just point out that I think discussing sides of what's linked can be interesting and relevant. In this instance I think discussing privacy around the amount of data Spotify stores is a relevant subdiscussion worth exploring. But complaints about their UI doesn't feel very relevant. Etc. Now that's only my opinion. What…
> But it's noise not disturbing a good signal. The signal will always be weak early on. Look at the comments now and it's clear that the upvote/downvote mechanisms are working sufficiently to address your concern. This should not be surprising. It's typically faster to not read the article and spew superficial off-topic comments than to write on-topic, substantive, and technical comments.
I came here expecting to read about the tech in the article or how others do big data processing stuff. Instead I get off topic Spotify rants.. Did you read the article or just see Spotify in the headline and decided your gripes therefore are relevant?
Agreed to I'll bite... > The intuition is that for datasets commonly and frequently joined on a known key, e.g., user events with user metadata on a user ID, we can write them in bucket files with records bucketed and sorted by that key. By knowing which files contain a subset of keys and in what order, shuffle becomes a matter of merge-sorting values from matching bucket files, completely eliminating costly disk and…
Before worker nodes had as much memory as they have now, almost everything needed to use small buffers and spill to disk. BDB (Berkeley DB) was an extremely common tool for doing out of core data operations. Because the ETL tools I was writing needed to run on machines with 512MB of ram, it required out of core algorithms. We easily had jobs processing 10-20GB with only 512M of ram.
I am sure I am missing something, reading the paper now.
http://kth.diva-portal.org/smash/get/diva2:1334587/FULLTEXT0...
Earlier quoted context omitted.
This is the most basic pattern for distributed joins - you hash on the join key in both tables and shuffle data based on hash ranges. In some systems like Redshift you can designate the key for distribution so that "related" records are already co-located on a single shard. > our data would already be sorted for the given keys (because within a partition Kafka has sorting guarantees) It's been a while since I used Ka…
Yes I guess my point is when using Kafka in combination with Kafka Streams and you produce things partitioned in a way that you need them for consumption then you do not need to do any shuffling in the instance where you want to join because data is already partitioned correctly.
I'm learning proper data flow in real time as I look to transition ETL of product data into Postgres to a more applicable system.
Finding the right learning resources is difficult! Cheers.
My off-topic rant: I'd really wish Spotify would focus on improving the core player experience. It has barely seen any improvements in years. * Not overwrite/delete my listening history everytime I switch devices * Allow tabs, or some way to resume what I've been listening to in different contexts * Option to open only one instance, instead of having multiple instances that mess with each other * Playing local files…