Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How good is Google’s Instant Mix?

musicmachinery.com

Post reply on HN
Print view View original

21–30 of 30 posts

Re: How good is Google’s Instant Mix?

#21

Post by shrikant »

kefs wrote: post #10
brianwhitman wrote: post #9

Earlier quoted context omitted.

Fair, but a few points: - Genius was much better than this when it launched. It's even better now, but it wasn't as bad back then as Instant Mix is today. - You don't think google has more data about music than Apple did when it launched Genius? YouTube, Music Onebox, search traffic. - (severe bias alert) EN's playlist APIs are the same excellent quality today as the day it launched (sept '10.) We're roughly 0.10% Go…

I'll absolutely agree that Google Music Instant Mix is nowhere where it needs to be.. not even close, but I still can't shake the feeling that this review is more of a "Please Buy Us" post... Just curious, in regard to the quality of EN since launching.. You guys we're clearly working on your algorithm for several years prior to launch[1] in late 2010.. That's sort of a warm-up period, no? [1] second last paragraph o…

I still can't shake the feeling that this review is more of a "Please Buy Us" post...

..so? I'd honestly like to see more of this sort of advertising.

Re: How good is Google’s Instant Mix?

#22

Post by tintin »

I think it would be more interesting to recommend music by mood instead of genre.

Jazz doesn't mean a thing. When I'm happy I don't want to listen to down jazz although it's jazz. Maybe I would like some happy funk and metal as well.

Re: How good is Google’s Instant Mix?

#23

Post by plamere »

chapel wrote: post #3

He addresses this in his article, but you really can't evaluate competitors to your own service in such a manner. Not only were the testing criteria highly subjective, the fact that songs he considered 'WTF' in the lists were the negative marks. I know music is a very subjective subject, I really think he could have been more objective about the whole thing. Maybe using other peoples music collections and getting the…

Hi Chapel - you suggest I could have been more objective, perhaps using other people's music collections and getting their own personal opinions. That, of course, would still be a subjective evaluation. It would be better, of course, more opinions means more data. In fact, I welcome people to make the same evaluation with their own collections. Enroll them in all 3 systems, create some playlists and evaluate them. Since most people don't have access to Google Music yet, this is hard to do. Still, you can look at the playlists that I generated and make your own WTF opinions about them. Or better yet, count the WTFs in the playlist Google created during the Google I/O keynote. You can see it at 28:29 of this - http://www.youtube.com/watch?v=OxzucwjFEEs Here's a screencap. https://skitch.com/plamere/r9x2k/youtube-google-i-o-2011-key...

There's no objective evaluation of playlists. I've proposed a simple, subjective one that I think gets the job done. I'm happy to try other ones if you have something to suggest.

Re: How good is Google’s Instant Mix?

#24

Post by brianwhitman »

lordlarm wrote: post #20
MikeCapone wrote: post #8

Could this be because this service is so new that they haven't yet had the benefit of gathering lots of data and stats from users?

I too believe so. Let Google build their product first, it is still a beta, for a reason. Music recommendation is close to impossible without lots of data - and it becomes unfair to compare Google Music with iTunes Genius, which has been on the market in 3 years.

(again, i'm the co-founder, grain of salt time, but check it out yourself) but The Echo Nest makes amazing similarity judgments and playlists without any of the data that you think Google needs to wait for. We crawl the web and analyze the audio and figure out what people are saying about music and what it sounds like. We don't need usage data at all to make recommendations. No usage data went into the EN results in the OP.

Re: How good is Google’s Instant Mix?

#25

Post by Tyrannosaurs »

kelnos wrote: post #14
omaranto wrote: post #11

Earlier quoted context omitted.

Nitpick: an empty playlist has no songs out of place, so it should score 0 WTFs (I don't understand how you got length(empty)=24). Giving Apple some WTFs instead of giving it 0 is clearly more in the spirit of the test than following the criterion to the letter would have been.

While that might make sense in using a logical definition of what a "WTF" is in this context, assigning 0 WTFs for the inability to generate a playlist is useless for the purposes of this comparison. From my perspective, I'd say an empty playlist is worth 24 WTFs, in the sense of "WTF, I expected 24 similar songs and got 0!"

I disagree.

Would you rather be given no results or bad results? I agree that no results isn't good but I'd rather an application that actually identified it couldn't do a good job than one that just threw a load of junk at me.

I think a score of 50% or so is probably about fair. It shouldn't get a good score certainly but I'd certainly rate it higher than one which just produced nonsense.

Re: How good is Google’s Instant Mix?

#26

Post by Tyrannosaurs »

nl wrote: post #16
chapel wrote: post #3

He addresses this in his article, but you really can't evaluate competitors to your own service in such a manner. Not only were the testing criteria highly subjective, the fact that songs he considered 'WTF' in the lists were the negative marks. I know music is a very subjective subject, I really think he could have been more objective about the whole thing. Maybe using other peoples music collections and getting the…

Paul has done a lot of work in the field of playlist evaluation. I think that if you survey the researchers in the field, the "WTF test" would be considered fairly reasonable - especially for a quick-and-dirty evaluation. Can you point to any specific songs that he said were WTF's and you think aren't, or vice/versa? If not then it would appear to meet the objectivity criteria. Using his own music collection might be…

I think you can argue the individual songs but the overall findings are sound - Google is poor, EchoNest is good, iTunes is good bar one song (which is suspicious). The methodology is a bit finger in the air but then so are the findings.

Re: How good is Google’s Instant Mix?

#27

Post by Tyrannosaurs »

Anyone else curious about why the Beatles returned nothing on iTunes?

[Note: All this applies to Beatles tracks ripped from CD rather than purchased via iTunes.]

I've just tried Genius (updated immediately before) on a selection of Beatles tracks in iTunes - Eleanor Rigby, Yellow Submarine, Lucy in the Sky with Diamonds, Sgt. Peppers Lonely Hearts Club Band and not one of them produces a playlist.

Even before they were selling Beatles tracks enough people will have ripped them from CD to have the data to produce playlists so it seems unlikely (but not impossible) that it's data related, which suggests that it's either an odd, very specific error or intentional.

What's stranger is that for some (but not all) of the tracks it will make Genius recommendations to buy...

Anyone any ideas? Some strange part of the Beatles licensing deal perhaps though that would be very odd as really who gains anything by that?

Re: How good is Google’s Instant Mix?

#28

Post by kefs »

DaveMebs wrote: post #19
kefs wrote: post #10

Earlier quoted context omitted.

I'll absolutely agree that Google Music Instant Mix is nowhere where it needs to be.. not even close, but I still can't shake the feeling that this review is more of a "Please Buy Us" post... Just curious, in regard to the quality of EN since launching.. You guys we're clearly working on your algorithm for several years prior to launch[1] in late 2010.. That's sort of a warm-up period, no? [1] second last paragraph o…

You could call it a warm-up period, but the point is that period is pre-launch. I think it is fair to assume a team at Google worked on this project before launch as well. I don't know how long they did for, but they could have chosen to take the time to develop a better algorithm prior to launch as well. (Note: Google's historical use of the Beta tag, has made it effectively meaningless IMO.) For whatever reason the…

It's quite a warm-up period.. Founded in 2005, first API went public in March 2008.

http://techcrunch.com/2008/03/27/first-machine-listening-api...

Re: How good is Google’s Instant Mix?

#29

Post by brianwhitman »

kefs wrote: post #28
DaveMebs wrote: post #19

Earlier quoted context omitted.

You could call it a warm-up period, but the point is that period is pre-launch. I think it is fair to assume a team at Google worked on this project before launch as well. I don't know how long they did for, but they could have chosen to take the time to develop a better algorithm prior to launch as well. (Note: Google's historical use of the Beta tag, has made it effectively meaningless IMO.) For whatever reason the…

It's quite a warm-up period.. Founded in 2005, first API went public in March 2008. http://techcrunch.com/2008/03/27/first-machine-listening-api...

Few things -- yes, we have been working on our stuff for a while. I've been doing it since 1999 at various grad schools and research labs, for example. Notice however we do not release things before they are good. I've spent 12 years working on this stuff and the state of the art simply wasn't ready until recently. First impressions are very important for music, if you lose the trust of your users it's hard to get it back.

Re: "warmup": My point was technologies like ours don't need a warm up period where users give us catalog and preference data. You can have great results without relying on that. Even better, we think.

Also, re: "please buy us" -- come by the office sometime and have some drinks and you'll see why we all found that comment pretty funny. Hopefully someone still reading this thread knows us well enough to +1 this. Come to a music hack day if you're not near boston.

(post-edit: davemebs -- our stuff is pretty easy to try for yourself if you're a developer. see developer.echonest.com . if not, wait a little bit for some consumer facing things. we're primarily developer-focused at the moment)

Re: How good is Google’s Instant Mix?

#30

Post by MikeCapone »

brianwhitman wrote: post #24
lordlarm wrote: post #20

Earlier quoted context omitted.

I too believe so. Let Google build their product first, it is still a beta, for a reason. Music recommendation is close to impossible without lots of data - and it becomes unfair to compare Google Music with iTunes Genius, which has been on the market in 3 years.

(again, i'm the co-founder, grain of salt time, but check it out yourself) but The Echo Nest makes amazing similarity judgments and playlists without any of the data that you think Google needs to wait for. We crawl the web and analyze the audio and figure out what people are saying about music and what it sounds like. We don't need usage data at all to make recommendations. No usage data went into the EN results in…

Fair enough, but if their technique is based on mining data from users of the service, then let's wait a little while and do the comparison again.
Post reply on HN