Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Didyougogo – An Altavista slayer

didyougogo.com

Post reply on HN
Print view View original

41–50 of 131 posts

Re: Show HN: Didyougogo – An Altavista slayer

#44

Post by gary__ »

misterman0 wrote: post #22
tcmb wrote: post #10

Kudos for your courage to make your great ambitions public from the start. 1. Does the site do any crawling on its own, or is the public index only fed from submissions? 2. It appears Umlaut/Unicode handling needs some work: When I search for "Käse" (German for 'cheese'), I get the response "0 results for 'Käse' in 'www' (0 ms)". At this point I'm not sure if there's actually 0 results or if it was actually sear…

Thanks! 1. You may submit a page. When I have a little more capacity that just 1 CPU/1 GB RAM I will also crawl. 2. I'll look into it. Thank you.

Would the common crawl dataset be useful to you starting out?

http://commoncrawl.org

Re: Show HN: Didyougogo – An Altavista slayer

#46

Post by tlb »

misterman0 wrote: post #21
spchampion2 wrote: post #7

I searched for "Cnn" and got 0 results. I searched for "Amazon" and got a five random results, including the IMDB page for "Rambo, Part 2." If this were really like AltaVista, I'd get 3 trillion results and have to use advanced Boolean logic to cut that down to the most useful 7,000 - so I guess having no results is sort of easier...

My boolean logic is here: [1] I'm sure it has flaws. Since the index had only five or six entries a couple of hours ago I set the matching to be wide instead of narrow. I'm also experimenting with loading the model with phrases, phrases and words or words only. I might have f-ed up the query parsing because of that. Remember, this is 0.1, fresh out of the press. Searching the tree is here: [2] Tokenization is here: […

Major search engines test every release against a list of search queries. You could start with https://trends.google.com/trends/topcharts. You should have an automated test script with a list of (query, good URL) pairs and make sure the good URL appears in the top few results.

Re: Show HN: Didyougogo – An Altavista slayer

#47

Post by scruffyherder »

kps wrote: post #25
keketi wrote: post #16

The minimalistic layout is a pleasure to use compared to AltaVista's bloated UI.

Altavista was great when its raison d'etre was to show off the Alpha. (I still miss proper boolean queries.)

There is a hack to have the desktop Altavista search tool index gopher...

https://blog.benjojo.co.uk/post/building-a-search-engine-for...

I've done it with a static set of data, the UTZoo Usenet data...

http://altavista.superglobalmegacorp.com

Shame it died on the vine, distributed, and curated search was a powerful tool in the days of Veronica and Archie

Re: Show HN: Didyougogo – An Altavista slayer

#48

Post by misterman0 »

pebers wrote: post #37

Definitely some ambitious goals. There's nothing bad about that, but this has an awfully long way to go - e.g. searching for "hacker news" works fine, searching for almost anything else didn't find anything relevant. So while it's nice to say it can run in 1CPU / 1GB, I'm not sure it's very useful at that size (but I don't know how big it'd have to get to "break even" there). Anyway, noted that it's a very early vers…

Thank you!

Yep, I have probably messed up the relevancy a bit because of constantly experimenting with how to load the model/index. Right now I'm using phrases (sentences) as well as words, both extracted during the tokenization process. Initially I used only phrases because using the current 65K vector-space model that would match any word to any phrase containing that word. There are perhaps sideeffects of reinforcing each word like that.

"long way to go"

I don't think so. The real bitch was to figure out how to maintain a good representation of the language model on disk. How to update it. Remove data from it. Now I anticipate a couple of months fine-tuning the balancing of the tree and testing relevance. From what I have heard so far, relevance is a little sub-par.

Scaling is the next thing. I have a great plan for that of course, mentioned somewhere in this thread.

Re: Show HN: Didyougogo – An Altavista slayer

#49

Post by misterman0 »

tlb wrote: post #46
misterman0 wrote: post #21

Earlier quoted context omitted.

My boolean logic is here: [1] I'm sure it has flaws. Since the index had only five or six entries a couple of hours ago I set the matching to be wide instead of narrow. I'm also experimenting with loading the model with phrases, phrases and words or words only. I might have f-ed up the query parsing because of that. Remember, this is 0.1, fresh out of the press. Searching the tree is here: [2] Tokenization is here: […

Major search engines test every release against a list of search queries. You could start with https://trends.google.com/trends/topcharts . You should have an automated test script with a list of (query, good URL) pairs and make sure the good URL appears in the top few results.

As preparation for this demo, yes I absolutely should have run such a test. Eagerness won.

Thanks for the link.

Post reply on HN