Live data from Hacker News

Google should open source what actually matters: their search ranking algorithm

cdixon.org

51–60 of 70 posts

Re: Google should open source what actually matters: their search ranking algorithm

#51
I am disturbed by the lack of faith in open source among the readers here. I mean, I understand some skepticism but this, to me, is pretty extreme IP-fetishism.

It's perfectly understandable. Lots of money is made in the existing paradigm and so many of you have incentive to believe it's the most rational system. Besides, hard to imagine Google's doing something wrong, right?

But the argument that "the only people this will help is spammers"? Really? Are the only people helped by open cryptographic protocols those who look to decipher traffic?

"They have every right to keep it secret and profit off of it." Let's allow this for a moment. We've still got a false dichotomy. Making it public will not deprive them of their profit. First, incredible marketing coup. Does anyone else remember how much of Google's initial strength came from its rabid fans? This would do a lot of re-energize that base.

Second, are there going to be new startups which can effectively compete with Google, even given its algorithm? How are they going to provide a better experience, or serve the volume of traffic, or index a larger amount of pages with the same algorithm with fewer resources?

Third, who says they have to authorize competitors to use it? This is I think one of the more interesting points. They could release the algorithm but license it only for non-commercial use for anyone not themselves. Ridiculous? Because Microsoft would violate such a deal anyhow? Because the algorithm could simply be reimplemented? Perhaps. But it would be another roadblock.

Fourth, why do we continue to believe that Google's strength comes from proprietary code? Why do we not recognize that its strength comes from mindshare, user experience, and quality of execution? None of which would be negatively affected by releasing the source.

Perhaps Google would face heightened competition as a result of releasing its algorithm. Perhaps this would outweigh the benefits of doing so. But there would be significant good which would result, including public feedback which would make it significantly harder for spammers to be successful.

But I'm just a foaming-at-the-mouth open source radical. And I'm sure history is on your collective side. If there's anything we've learnt from a hundred years or so of Computer Science (overestimate if you only count actual programming, underestimate if you allow Babbage as you should), it's surely that secrecy is what drives innovation, right?

Re: Google should open source what actually matters: their search ranking algorithm

#52
post #24
post #5

That is a completely ridiculous point to make. Come on, be serious here. What would be the benefit to anyone except for spammers, if Google open-sourced their algorithms? It's a lose-lose option. Google loses some money, we lose a lot of search engine accuracy, and spammers gain new ways to screw around with search results. And does anyone gain from it? Not really. At this point, Google's supremacy is due just as muc…

The stated goal isn't to help other search engines; it's to get the community to help Google improve, for the good of all its users. The theory goes: if you let more people look at the algorithm, they can suggest better answers than Google alone, to thwart the spammers who are gaming the system either way. But I suspect the real goal has more to do with the deep suspicion that Google can or does do a significant bit…

I'll believe that when I see open source achieve a conclusive victory against email spam.

Spam:email = 90% or worse

in search results it is closer to the opposite.

I'm sure once the open source community actually manages to do this and prove the 'many eyes' doctrine applies to more than just security and bug fixes that many doors will open.

Re: Google should open source what actually matters: their search ranking algorithm

#53
post #42

Earlier quoted context omitted.

Agreed. After reading the comments at cdixson.org by the author and reading openness article at Google, it seems like author is simply saying: "I agree they won't open it. But then they should stop talking sactimoniously about openness." which I agree with as well. Here is an interesting perspective: "And millions of websites will have to continue blindly relying on a small group of anonymous engineers in charge of t…

> Perhaps transparency is what Google needs the most at the moment. Absolutely, and not only on the search engine part of it, Adwords/Adsense are extremely opaque and as a publisher continuously give you the feeling you're being screwed without any chance to prove that for lack of information.

As a publisher, couldn't you do some sort of A/B testing with another ad provider? Something like 10% of page views using another set of ads? Or, maybe setup some kind of onclick tracking to verify ad clicks?

I don't know what could be done as verification for the advertiser, but the publisher should be able to do something, right?

Re: Google should open source what actually matters: their search ranking algorithm

#55
post #14
post #2

The notion that being 'committed' to open source means that a company should give away everything of value is ludicrous. Google came up with great software that makes millions of peoples' lives easier, and they have every right to keep it secret and profit off of it. The zealous idea that open source and trade secrets cannot coexist is detrimental to the open source movement as a whole, in my opinion.

Well put. Almost any good idea taken to its extreme is a bad idea.

High taxation eventually reaches communism, if there's benefits like healthcare 'refunded' to the citizen and if there isn't then that ends up being a kleptocracy. Low taxation eventually reaches anarchy as the government is no longer in control.

Increasing birth rates enables economies to grow without becoming stagnant and old, placing undue demand due to pensions. An exceptionally high birth rate creates food production problems, housing problems, and generally the 'sardine can' effect. Low birthrates create a stagnant civilization that will eventually reach a tipping point when taxes get so high to pay for pensions that the younger generations will protest.

Market freedom to the extreme leads to monopolies taking advantage, 'gentlemen's agreements' between companies to price fix and many other problems that rules and regulations try to control. Equally a controlled market develops problems in and of itself as it's unable to respond naturally to market fluctuations, and quickly becomes unruly and unstable, and as many of the countries who have held long-term controlled markets are testaments to, they develop massive black-market economies to deal with these.

The zealous burning of fossil fuels, aside from global warming concerns, can expel vast quantities of radioactive materials, sulfurous emissions and a long list of things we've accepted in western society as being more harmful than smoking to the wide populations health, but rules and regulations on air quality increasingly get more lax (at least they have been where I live). With zero fossil fuel production, however, our civilization would collapse (at least with an immediate stop to production), and I'm not just talking the halting of transportation vehicles, I'm talking the stop in production of the advanced polymers used in virtually all aspects of our lives. The majority of people wouldn't have clothes on their back . . . my wardrobe would certainly be a lot cheaper if my clothes were all genuine cotton and wool. Oh and farewell cheap elasticated sweatpants.

Every good idea taken to its extreme is a bad idea and will create monumental problems in its wake. Electric vehicles are amazing, however a pure-electric is a ridiculous prospect in winter when the capacity hits below 60% of room-temperature, suddenly your vehicle can't get you to work and back on a single charge. Zero integration of hybrids is, as everyone knows, damaging to the environment but also an extreme waste of entropy. A 1 tonne vehicle at 100km/h is a massive store of energy, and allowing that to be wasted is criminal, the amount of energy humanity has wasted on braking at highway speeds would likely run the entire human civilization for over a year.

Re: Google should open source what actually matters: their search ranking algorithm

#56
post #42

Earlier quoted context omitted.

Agreed. After reading the comments at cdixson.org by the author and reading openness article at Google, it seems like author is simply saying: "I agree they won't open it. But then they should stop talking sactimoniously about openness." which I agree with as well. Here is an interesting perspective: "And millions of websites will have to continue blindly relying on a small group of anonymous engineers in charge of t…

> Perhaps transparency is what Google needs the most at the moment. Absolutely, and not only on the search engine part of it, Adwords/Adsense are extremely opaque and as a publisher continuously give you the feeling you're being screwed without any chance to prove that for lack of information.

You say that now but if they provided that transparency then the amount of fraudulent activity would skyrocket. Knowing the details of the algorithms involved does nothing to help you as an advertiser or searcher. Googles extreme impartiality in search and ads is their greatest strength and open sourcing the algorithms involved would do nothing to increase that strength and contribute to a great deal less useful ads and search results on average.

Re: Google should open source what actually matters: their search ranking algorithm

#57
post #56

Earlier quoted context omitted.

> Perhaps transparency is what Google needs the most at the moment. Absolutely, and not only on the search engine part of it, Adwords/Adsense are extremely opaque and as a publisher continuously give you the feeling you're being screwed without any chance to prove that for lack of information.

You say that now but if they provided that transparency then the amount of fraudulent activity would skyrocket. Knowing the details of the algorithms involved does nothing to help you as an advertiser or searcher. Googles extreme impartiality in search and ads is their greatest strength and open sourcing the algorithms involved would do nothing to increase that strength and contribute to a great deal less useful ads…

> Googles extreme impartiality in search and ads is their greatest strength

Without proof that is nothing but handwaving, if they are as impartial as you suggest they are they've got nothing to lose by letting you know what their take is and how big a portion of their clicks are confirmed fraud.

Re: Google should open source what actually matters: their search ranking algorithm

#58

Earlier quoted context omitted.

> communism being one of the more spectacular failures of open Wait, what? Communism is a spectacular failure of centralized planning over the entire economy. It has nothing to do with open or closed in principle, and in every implementation it has been aggressively closed in terms of information flows and decision making.

> Communism is a spectacular failure of centralized planning over the entire economy. Wow, and here I was thinking it was a form of government. Communism in principle has absolutely nothing to do with the flow of information and/or decision making, and its implementations have only very little to do with its basic ideas. A true communistic government has never even been tried, we've only seen a 'new boss same as the…

I believe that Communism is largely an economic system with politics being aside effect of the economic model. For example, the opposite of Communism isn't democracy, it's capitalism. And actually, in the purest sense, Communism is completely about decision making. According to that bastion of scholarly thought Wikipedia:

    "Pure communism" in the Marxian sense refers to a classless, stateless and 
    oppression-free society where decisions on what to produce and what policies 
    to pursue are made democratically, allowing every member of society to 
    participate in the decision-making process in both the political and economic 
    spheres of life.
But that's one of those theory and practice things... great in theory, horrible in practice.

Re: Google should open source what actually matters: their search ranking algorithm

#59
This post manages to completely miss the point of the Google letter on openness.

As some have incorrectly asserted, Google's latest post is not about "everything open", it is an internal memo to product managers that it occurred to them would be fit for public consumption.

The central thesis of the letter is that "open" is hard to define (and defined differently by different people) and yet crucially important. Because it is hard to define, many people misuse it (OpenDNS comes to mind), and no one can call them on it because it is a vague term.

Internally, Google employees argue about what open means, and the purpose of this letter was to clarify what open means to Google. They lay it out in extremely clear terms, and it is well structured.

This response essay by Dixon misses all that completely - he unilaterally defines "open" to mean "open source", and then asks for Google to give away their software, claiming that if they don't, they're hypocrites.

Going back to Google's definition of "open" (which is complex), one significant aspect they emphasize is the notion of "lock-in", i.e. locking in users, and locking out competitors. This is one area where Google puts their money where their mouth is; they let users leave. You can get at your email, export your Google Reader content, download all your Google Docs, archive your Picasa pictures, and export all your contacts. That means if a competitor comes along with something better, you can start using it. Try that with Facebook.

Dixon acting like that is worthless because it isn't the "open-source" form of "open" is patently absurd. If this is confusing, I highly recommend reading Simon Phipps essays on open source to understand what kind of software makes a good candidate for a business to open source (operating systems, browsers, compilers...the basics that we all use to compete) and what does not (grid/cloud computing algorithms, specialized software, and other types of software that are differentiators). Essentially, the business model is to pool our collective resources in areas in which we don't want to compete (web server software, operating systems) and to spend time innovating in areas we do wish to compete (search algorithms, voice recognition, image analysis, etc.) Over time, as those specialized areas become more mainstream, they become better candidates for open-sourcing.

Re: Google should open source what actually matters: their search ranking algorithm

#60

Earlier quoted context omitted.

> communism being one of the more spectacular failures of open Wait, what? Communism is a spectacular failure of centralized planning over the entire economy. It has nothing to do with open or closed in principle, and in every implementation it has been aggressively closed in terms of information flows and decision making.

> Communism is a spectacular failure of centralized planning over the entire economy. Wow, and here I was thinking it was a form of government. Communism in principle has absolutely nothing to do with the flow of information and/or decision making, and its implementations have only very little to do with its basic ideas. A true communistic government has never even been tried, we've only seen a 'new boss same as the…

>Wow, and here I was thinking it was a form of government.

Like most ideologies, the term "communism" is gooshy enough to mean whatever you want it to mean - but at heart, communism is concerned with the matter of who owns and controls the means of production. It's a form of government insofar as the decision on who gets to control the means of production is ultimately a political decision.

Communists believe the people should own and control the means of production - but every real-world implementation has used government as a proxy for people.

I suppose it's possible to run a communist society via direct democracy on a very small, local scale; but good luck trying to scale that up to the national or even regional level. At some level you need to adopt some kind of more or less representative government. In the case of communism, the track record is definitely skewed toward the "less" end of the representation spectrum.

In other words, your claim:

>A true communistic government has never even been tried...

is a straightforward No True Scotsman fallacy. It's no different from "real" conservatives distancing themselves from the Bush administration because the implementation of their political system ended up violating most of its ostensible core values.

If you can't implement an ideology without discarding the very principles that define the ideology, it's a broken ideology.

Post reply on HN