Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wikipedia starts work on $2.5M internet search engine project to rival Google [pdf]

m.wikimediafoundation.org

Post reply on HN
Print view View original

141–150 of 192 posts

Re: Wikipedia starts work on $2.5M internet search engine project to rival Google [pdf]

#141

Post by rmc »

abalone wrote: post #38

Summary of the approach (p10): "1) Public curation mechanisms for quality; 2) Transparency, telling users exactly how the information originated; 3) Open data access to metadata, giving users the exact date source of the information; 4) Protected user privacy, with their searching protected by strict privacy controls; 5) No advertising, which assures the free flow of information and a complete separation from commerc…

> 4) Protected user privacy, with their searching protected by strict privacy controls;

They'll have to keep the servers outside the USA then. It's illegal for European organisations to transfer personal data to the USA now that Safe Harbour is invalid.

Re: Wikipedia starts work on $2.5M internet search engine project to rival Google [pdf]

#142

Post by bratch »

xojoc wrote: post #23
seven-dev wrote: post #19

Earlier quoted context omitted.

(Default nginx page showing up on your website)

It works for me. Care to show me a screenshot?

The host (assuming the same host) is responding with a different website when accessed via IPv4 vs IPv6.

  $ curl -4s http://kairos.xyz/ | grep title
      Kairos

  $ curl -6s http://kairos.xyz/ | grep title
  Welcome to nginx on Debian!

  $ host kairos.xyz
  kairos.xyz has address 107.161.29.121
  kairos.xyz has IPv6 address 2604:180:0:a54::24d9

Re: Wikipedia starts work on $2.5M internet search engine project to rival Google [pdf]

#143

Post by gmac »

noelsusman wrote: post #33
quickben wrote: post #18

Earlier quoted context omitted.

I run a search engine that indexes an entire western country for beer-worth of vps. I spent few months to build it though. I don't see scaling for the world to be hard given the money they have.

Building a basic search engine is relatively easy. Building one that rivals Google is extremely difficult, and not just because they're so big and convincing people to switch is hard. It's much easier to have good results when you know that the websites you're indexing don't care about you at all. Once you get popular enough to rival Google everyone and their mother will be trying to game you and that changes the pro…

There's another incumbent advantage here: I imagine it's much easier to provide good results when you also have data on which results thousands or millions of people clicked on for millions of search terms.

Re: Wikipedia starts work on $2.5M internet search engine project to rival Google [pdf]

#144

Post by infbtag »

selectodude wrote: post #116
frik wrote: post #103

Earlier quoted context omitted.

> We have duckduckgo already DuckDuckGo a meta-search-engine! It relies mainly on Yahoo Boss API which uses Bing search (for most countries)! Yahoo Boss API turned from free to expensive in early 2015 and the future of Yahoo (tech company, not Alibaba stock) is uncertain. We definitely need more search engines, only 6-7 exist that cover a wider range (international). Search on HN to retrieve the list, we had this dis…

BOSS is being discontinued on March 31st, so I imagine that DDG has some sort of idea what they're going to be doing after that date. Any insight?

They could use the Bing API directly or just stick with Yandex.

http://datamarket.azure.com/dataset/bing/search

Re: Wikipedia starts work on $2.5M internet search engine project to rival Google [pdf]

#145

Post by newscracker »

jarcane wrote: post #132
meowface wrote: post #77

Earlier quoted context omitted.

It's intentionally broken. Despite many real (though also some exaggerated) counter-examples, Facebook does have features to protect privacy. One of those things is you generally can't get very useful results from search unless you're friends with someone, or a friend of a friend (depending on the user's privacy settings). You can't see general trends, or even search for every person with a given first and last name…

It can't even find anything in my own posts . If it's a permission problem, that's a pretty fuckin' serious permission problem right there. On top of which, Graph Search is still disabled inexplicably in many regions, for reasons which were never explained.

> It can't even find anything in my own posts.

Ditto. I have seen this as a crippling deficiency in the Facebook platform. Not being able to search properly in my own posts or in the groups I'm a member of really sucks. For all the engineering prowess, open sourced tools, etc., shown by Facebook, the lack of a working search makes the company seem incompetent from the top down.

Sometime ago I started storing important information (like others links, my own comments, etc.) outside of Facebook where I can find them easily. I also started a Facebook page and added Notes into it to make it easier to document, find and share things. It seems ridiculous that I'd have to do this just to have access to information, but that's been the sad state for years.

Re: Wikipedia starts work on $2.5M internet search engine project to rival Google [pdf]

#146

Post by inaudible »

Anyone want to hazard a guess at the technology they plan to implement to get this started?

Surely this is not designed to be written from scratch, so..

- Are they using known lexical & semantic scanners? - Is it focused on English language first? - What crawlers will scan content? - I'll assume it's an open platform, but license for contributors? - What database architecture will hold the graph? - How does it know the mark of authority, and is this primarily based on human input learning or machine learning?

I'm sure $2.5M wont touch the sides, but maybe if it's a well directed project, with healthy user contribution, based on interesting technologies they might develop a good backbone architecture. Ambitious for sure.

Re: Wikipedia starts work on $2.5M internet search engine project to rival Google [pdf]

#147

Post by nl »

cpeterso wrote: post #137
luxpir wrote: post #131

Earlier quoted context omitted.

This may or may not be what you're looking for, but my freelance site just can't hit the front page in my industry when dozens of highly funded agencies can dominate it. I've published a 50k word industry-specific book on my site, have SEO'd as much as possible and have an older domain than those better funded. Won't link to it here, but it seems to be a real issue to me. If a search engine let you differentiate and…

btw, your site's redirect to the HTTPS version doesn't seem to work correctly in Firefox, Safari, or IE. After reading your comment, I was curious to learn more. When I typed in just your domain name plus CMD+ENTER (which adds "www." and ".com" to the address bar text in Firefox), I got a 404 page, not the 301 redirect to the HTTPS site. When I add " http://" , the redirect seems to work.

This.

I don't work in this area, but I'd say that 99% of the time I hear someone complaining about how Google is favoring sites that pay for advertising over them I find that they are making these incredibly basic errors.

For me, http://www.linguaquote.com/ gives a 404. It's only when I go to https://www.linguaquote.com/ that it works.

Re: Wikipedia starts work on $2.5M internet search engine project to rival Google [pdf]

#148

Post by araneae »

jcrben wrote: post #66

This was/is actually an extremely controversial project. The corporation (basically the Executive Director) pursued the grant and the idea without soliciting input or really disclosing it to the community of editors, and eventually one of the community-elected trustees was removed for questioning the lack of transparency. The community has a long list of software improvements that they'd like to see to the core platf…

> and eventually one of the community-elected trustees was removed for questioning the lack of transparency

This was a hypothesis about his removal when he was initially was removed, and has since been refuted by multiple sources.

Re: Wikipedia starts work on $2.5M internet search engine project to rival Google [pdf]

#149

Post by cJ0th »

tuvalie wrote: post #11
cmarschner wrote: post #4

This is a very, very, very small amount of money if you want to build a search engine, let alone one "to rival Google" (source?). Looks like the goals are realistic, though - look how wikipedia search could be extended beyond results from wikipedia.org, build some test sets. And get a better idea what it really is that is supposed to be built.

I run a knowledge engine project that actively mines facts from web-based sources and third-party data dumps. It was featured on the front page of HN a while ago, and has a total of $10 in funding (from a single donation; not a typo). I have, however, put a ton of time into it, and it's something I'm very passionate about. I'm fairly confident Wikipedia can have success making initial headway on their grant objective…

So apparently Einstein has his own, official website (einstein.biz). Who knew? It has got a proper shop and everything!

Treat yourself to this relatively expensive USB stick:

http://www.shopeinstein.com/mimobot-16gb-usb-flash-drive/det...

Re: Wikipedia starts work on $2.5M internet search engine project to rival Google [pdf]

#150

Post by tallanvor »

patrickaljord wrote: post #68
elmar wrote: post #59

Earlier quoted context omitted.

Facebook search is broken, twitter search works much better.

It's only broken because most people share with their friends only and search cannot expose their posts. This makes real time search way less efficient than twitter.

That's what security trimming is for. You index everything, including the permissions for each item and provide the searcher's permissions (whether it's an acl, a claim, or some other form) as part of the query. Anonymous or unauthenticated users get a "public" claim that only returns results available to everyone.
Post reply on HN