Live data from Hacker News

Sad but true: Napster '99 still smokes Spotify 2012

theregister.co.uk

61–70 of 94 posts

Re: Sad but true: Napster '99 still smokes Spotify 2012

#62
post #60

It is really hard to compare Napster and Spotify as they are entirely different beasts. Let me just say that I built the Napster server, so I'm biased. I have also been a Spotify user for several years. For me, Spotify is a sub par experience because I don't know what I want and I have different music tastes than my friends. The only thing that makes Spotify useful for me is sites like http://sharemyplaylists.com/ .…

Woah, you wrote the server for Napster?! That's awesome! Can you talk about that a bit? What was the underlying algorithm, what was the stack and how much data were you pushing at your max?

Which underlying algorithm?

The search engine was built on a ternary tree with a custom merging algorithm. I honestly don't know if the merging algorithm has a name as it was something I came up with (literally) while sleeping one night. Because we mostly used ID3 tags and file names, it was completely unnecessary to use a stemming algorithm because if you typed in a misspelled search, given the size of our index, there was probably someone who tagged their file using the same misspelling.

The network biasing code used BGP data combined from a number of looking glass servers to build a map of ip/prefix -> ASN number and ASN->ASN distances. It was then used to reorder search results based on network distance to users so they would bias towards their own networks and save ISPs money and speed transfers on broadband connections.

Servers were linked through a fully meshed network. Each had presence information about every user on the network so that they could route IMs around. The chat system was semi-linked (fully linked on some servers, but we couldn't fully link the whole thing because the client had no administrative functions for chat). If we couldn't send the user back enough results for a search, the query was simply passed around the backend.

The whole thing was written in C++. At its peak, there were about 2.3 million users online at any given time (80 million total users growing by a million every 4 days). The system would be indexing about 17.6 million files per second (and de-indexing about the same amount). The whole system pushed out about 2 Gbps of bandwidth in search results (which were tiny).

Napster was one of the very first services to push past 10K connections on a Linux machine. At peak, I could get over 100K users on a single process (though I'd run out of memory indexing files on the tiny 2 GB machines and blow out the NIC sending search results). During normal operations, each server process had around 40K users on it and between 7-12 million files indexed.

There were a bunch of side infrastructure things no one saw. Court mandated copyright filtering systems, recommender systems (most for play), load balancing servers, bot detection and sequestration systems, analytics reporting jobs, etc.

Nowadays, I could probably fit all of Napster on one big machine. Heh.

Re: Sad but true: Napster '99 still smokes Spotify 2012

#63

It is really hard to compare Napster and Spotify as they are entirely different beasts. Let me just say that I built the Napster server, so I'm biased. I have also been a Spotify user for several years. For me, Spotify is a sub par experience because I don't know what I want and I have different music tastes than my friends. The only thing that makes Spotify useful for me is sites like http://sharemyplaylists.com/ .…

Have you not used any of the Spotify Apps (e.g. MoodAgent, Last.fm, and, of course, sharemyplaylist)? If not, then I highly recommend you do - be interesting to see if the experience changes your opinion.

Seems to me that the Spotify API opens up all sorts of possibilities for music discovery.

Re: Sad but true: Napster '99 still smokes Spotify 2012

#64
post #60

Earlier quoted context omitted.

Woah, you wrote the server for Napster?! That's awesome! Can you talk about that a bit? What was the underlying algorithm, what was the stack and how much data were you pushing at your max?

Which underlying algorithm? The search engine was built on a ternary tree with a custom merging algorithm. I honestly don't know if the merging algorithm has a name as it was something I came up with (literally) while sleeping one night. Because we mostly used ID3 tags and file names, it was completely unnecessary to use a stemming algorithm because if you typed in a misspelled search, given the size of our index, th…

Wow, that's incredibly fascinating! Thanks for posting such a great reply! You built something that really changed the world, in my opinion, and you should be very proud.

I'm particularly pleased by the use of ASN/distancing weights in your results! None of the BT trackers I've hacked around on have had anything like that in them.

Where is the code now? It should be in a museum.

Re: Sad but true: Napster '99 still smokes Spotify 2012

#65

It is really hard to compare Napster and Spotify as they are entirely different beasts. Let me just say that I built the Napster server, so I'm biased. I have also been a Spotify user for several years. For me, Spotify is a sub par experience because I don't know what I want and I have different music tastes than my friends. The only thing that makes Spotify useful for me is sites like http://sharemyplaylists.com/ .…

Have you not used any of the Spotify Apps (e.g. MoodAgent, Last.fm, and, of course, sharemyplaylist)? If not, then I highly recommend you do - be interesting to see if the experience changes your opinion. Seems to me that the Spotify API opens up all sorts of possibilities for music discovery.

I've been using last.fm since it was audioscrobbler. I wrote one of the mac scrobblers years ago to upload my iPod plays. While it is fun to track my analytics, its user recommender is poor at best and the radio, I don't know, it just doesn't match my taste. I use it all the time to find popular songs by artists I've just discovered though. Great for evaluation.

Sharemyplaylist.com is basically what makes Spotify useful for me at all. I still can't talk to anyone, but at least I can have a playlist that sounds roughly like the songs belong together. I'll rarely discover new music on it.

For music discovery, I find http://hypem.com and mp3 blogs to be far more useful. Still no strong community, but at least I actually discover new interesting things.

Re: Sad but true: Napster '99 still smokes Spotify 2012

#66
post #64

Earlier quoted context omitted.

Which underlying algorithm? The search engine was built on a ternary tree with a custom merging algorithm. I honestly don't know if the merging algorithm has a name as it was something I came up with (literally) while sleeping one night. Because we mostly used ID3 tags and file names, it was completely unnecessary to use a stemming algorithm because if you typed in a misspelled search, given the size of our index, th…

Wow, that's incredibly fascinating! Thanks for posting such a great reply! You built something that really changed the world, in my opinion, and you should be very proud. I'm particularly pleased by the use of ASN/distancing weights in your results! None of the BT trackers I've hacked around on have had anything like that in them. Where is the code now? It should be in a museum.

No problem! It occurred to me that I've never really talked about the Napster architecture or algorithms publicly.

I once hacked up Transmission to re-prioritize based on network distance. It worked rather well when it discovered over the DHT. It is a lot easier when you only have to calculate distance between you and other networks. Storing the graph of distances between two arbitrary users is harder.

The code was part of the assets that were bought out of bankruptcy by Roxio (who renamed themselves Napster). I doubt it is being used for anything.

Re: Sad but true: Napster '99 still smokes Spotify 2012

#67
post #23

The fatal flaw of Spotify, Rdio, and even Last.fm is that finding good music is just too much work. Too much social and too many clicks. I wish Pandora could buy one of these services and combine their discovery engine with the catalogues of the subscription services. Until then, I think I'll just keep buying music.

Hey, if you like discovery try Audiogalaxy Mixes - we just launched it a week or so ago [1]. We think it's better than Pandora (much larger catalog, smarter algos, more input types) - tell me if it's not. It's free on the web (no ads) with a free trial on mobile, after which we charge a small fee to cover licensing costs. If you email viraj@audiogalaxy.com with your account I'll comp you a free month. I'd love to kno…

Audiogalaxy looks really good, but I can't find a radio mode. (like last.fm, where you enter a tag / artist and it plays similiar songs) Is there such a thing?

Re: Sad but true: Napster '99 still smokes Spotify 2012

#68
post #18

It's interesting that it's more than a decade since Napster, and the music industry hasn't gone bust yet. It's almost like P2P cannot destroy the main music industry.

no but the music industry can destroy the music industry, and they're doing a bang-up job of it. maybe a different way to think of it is: it's been more than a decade since napster and the music industry is just now beginning to get a clue.

The iTunes Music Store has been around since 2003. It has been three years since all DRM has been removed and files are 256kbit/s AAC.

Granted, this happened despite the music industry's efforts, it took a lot of pressure from Apple to get there. But still, I very much prefer the current situation over when Napster reigned. (Does noone remember the amount of garbage on Napster? The malware, the fake files, the low quality files? Not to speak of the terrible speeds -- many peers were still on dial-up...)

With iTunes, you can find new music easily, download high quality files, and get the album art and liner notes. Downloading and purchasing is fast and safe. You get to pay a fair price and know that the rights holders are being compensated.

Re: Sad but true: Napster '99 still smokes Spotify 2012

#69
post #27

Earlier quoted context omitted.

But they have already lost that power and are never going to get it back again.

SOPA proves they don't agree with you, unfortunately.

It doesn't really matter in the long run what they choose to believe, given that it is essentially just a game of coders vs lawyers that is being played out on a global computer network.

All SOPA did was underline yet again how little they understand the actual technology involved.

Re: Sad but true: Napster '99 still smokes Spotify 2012

#70
post #18

It's interesting that it's more than a decade since Napster, and the music industry hasn't gone bust yet. It's almost like P2P cannot destroy the main music industry.

I think in retrospect, it destroyed the music industry model, the music industry just doesn't know it yet.

I'm saving this whole thread re-reading it over and over again, thanks to Aloisius chiming in (giving a valuable and rare peek at the back-end), and the insight into what drives music discovery.

I do credit Napster for awakening a love for music. My father had a deep record collection from the 60's, and he had mentioned some very hard to find items he's wanted for decades that I found on Napster in couple of hours of searching. This was what made it awesome for me. The ability to browse someone else's collection was an incredibly effective music discovery system, and introduced me to music I'd never have discovered on my own.

Post reply on HN