Show HN: Didyougogo – An Altavista slayer • Hacker News Board

Re: Show HN: Didyougogo – An Altavista slayer

#91

Post by medecau » Sun, Aug 12, 2018, 6:09 PM UTC

I thought of something similar has a holiday project. A small search engine using SQLite FTS5 for a small set of websites crawled with Scrapy.

I made it public yesterday on https://fts.fail/

Good luck slaying that dragon though.

Re: Show HN: Didyougogo – An Altavista slayer

#92

Post by mamurphy » Sun, Aug 12, 2018, 6:19 PM UTC

vnglst wrote: post #83

oh-moses wrote: post #54

Awesome, but man, that name really needs some work. It sounds like I'm asking a two-year-old whether he's been on the potty.

Drop the 'did'? Much cleaner. And maybe also the 'you'?

Maybe rearrange the letters in gogo and add le to the end also?

Re: Show HN: Didyougogo – An Altavista slayer

#93

Post by asaibx » Sun, Aug 12, 2018, 6:22 PM UTC

As others have commented, love the ambitiousness of this! However, Unicode searches do not seem to work at all -- not just "中文" but also even "français" gives an error. Unicode support is something you definitely want to build in from the very beginning in order to avoid headaches (for you and users) in the future. Even if there is no content in the index, the presence of non-ASCII characters in the search term should not lead to a server error. Suggest you make Unicode the default encoding for everything even if you are not planning on supporting non-English search results for the moment, just to avoid unexpected errors when people search for things like "café" for example.

Re: Show HN: Didyougogo – An Altavista slayer

#94

Post by actuallyalys » Sun, Aug 12, 2018, 6:23 PM UTC

0xcde4c3db wrote: post #59

albertgoeswoof wrote: post #8

Earlier quoted context omitted.

The challenge is not building the search engine, it’s in building the index. That’s why google wants every drop of data

I've sometimes wondered whether it would work for a search engine to reduce the indexing problem by focusing more on quality than quantity. Rather than indexing everything in the universe and then trying to rank it, focus on maximizing ROI and keeping the aggregate quality of the corpus up by aggressively pruning low-quality paths up-front. In practice this might require splitting the difference between classic Yahoo…

You can sort of simulate this by searching discussion sites like Hacker News or Reddit. There's no pruning, but users do vote on what's most interesting or relevant to them. I find searching HN is useful when I'm looking for tools designed in the way HN readers tend to like: command-line, open source, and using a standard format.

Re: Show HN: Didyougogo – An Altavista slayer

#95

Post by usermac » Sun, Aug 12, 2018, 6:23 PM UTC

What just happened? I search for a park I visited just yesterday. "186" hits(?) and two of those were two top page HN sites I just visited!? I'm spooked.

Re: Show HN: Didyougogo – An Altavista slayer

#96

Post by pferdone » Sun, Aug 12, 2018, 7:00 PM UTC

Searched for „warez“... didn‘t return anything... I want to live in the old days again :‘(

Re: Show HN: Didyougogo – An Altavista slayer

#97

Post by mkl » Sun, Aug 12, 2018, 7:01 PM UTC

Re: Show HN: Didyougogo – An Altavista slayer

#98

Post by azinman2 » Sun, Aug 12, 2018, 7:09 PM UTC

misterman0 wrote: post #88

azinman2 wrote: post #78

Got zero relevant results. Not even sure how the results came back, as the words weren’t in there. Tried “Taoist tai chi,” then “Taoism”. Love the ambition, but a long way to go go.

Thank you and sorry about that. Feel free to submit a suitable page about taoism.

I’m the wrong one to submit as I’m trying to learn more myself. Wikipedia entries are an easy place to start, and that should be straightforward to add to your index.

I think it’s problematic to have random people submit to the index with no incentive. I’m just becoming interested in tai chi, but I run no such webpage (who usually submits). There might be a way to gamify or otherwise incentivize people, but that’s a very non-scalable approach. Really only automated crawling can be done to significantly widen your index. It’s just very resource intensive... but good luck! I hope you can go far!

Re: Show HN: Didyougogo – An Altavista slayer

#99

Post by sleepychu » Sun, Aug 12, 2018, 7:22 PM UTC

This is neat & impressive!

Why would I use this over duckduckgo? (Assuming that we're some time on and the index is comparable?)

Re: Show HN: Didyougogo – An Altavista slayer

#100

Post by scruffyherder » Sun, Aug 12, 2018, 7:32 PM UTC

nl wrote: post #69

scruffyherder wrote: post #47

Earlier quoted context omitted.

There is a hack to have the desktop Altavista search tool index gopher... https://blog.benjojo.co.uk/post/building-a-search-engine-for... I've done it with a static set of data, the UTZoo Usenet data... http://altavista.superglobalmegacorp.com Shame it died on the vine, distributed, and curated search was a powerful tool in the days of Veronica and Archie

istributed, and curated search was a powerful tool in the days of Veronica and Archie No it wasn't. I'm old enough to have (tried to) use it, and it was terrible. It was usually quicker and got better results to manually connect to FTP sites, and run directory listing on likely directories untily ou found what you were looking for.

I'm old enough to have used it as well, and was able to find things with it..