51–60 of 131 posts
Earlier quoted context omitted.
Thanks! 1. You may submit a page. When I have a little more capacity that just 1 CPU/1 GB RAM I will also crawl. 2. I'll look into it. Thank you.
Would the common crawl dataset be useful to you starting out? http://commoncrawl.org
Interesting idea. Isn't it a little late to slay Alta Vista though? :) I searched for apple. Top result was the archive.org macos that showed up here on HN recently, 2nd and 3rd were apple.com indexed 10s apart. Then some odd results - though they do include the word apple on page just once. The imdb page for 12 Monkeys appears 3 times. I guess you're not trimming duplicates? Seems like you need some way to weight ra…
I "googled" Google using Did yougogo, and Google.com didn't appear once on the first page. Funny.
Did you submit both a query and a URL?
Did you go go?
A search engine without https, I think I'll stick with Google for now.
Thanks.
when i submit something to the search engine, it produces a result that doesn't have anything to do with the search term. it's unclear to me how i am supposed to help improve this.
There is something wrong currently with relevance, probably because of query parsing errors but perhaps also in how text is tokenized. This whole idea revolves around relevance so this is of course embarrassing. But it's 0.1 alpha. And it _did_ work on my machine.
Thanks for trying.
Earlier quoted context omitted.
Major search engines test every release against a list of search queries. You could start with https://trends.google.com/trends/topcharts . You should have an automated test script with a list of (query, good URL) pairs and make sure the good URL appears in the top few results.
As preparation for this demo, yes I absolutely should have run such a test. Eagerness won. Thanks for the link.
But for some crazy reason, I kinda like this. It feels like the 90s internet. The links included so far have that same random mix of lots of nerdy links, homepages & personal blogs, a few religious sites, and the occasional big news website. Because there's no crawler yet, it's limited to the specific pages people thought were noteworthy. And because the index is so limited, I'm stumbling on interesting things.
It's so weird looking at this and thinking "Y'know, maybe this could also work if the links were curated into yet another hierarchical officious oracle", or "if this site let me pay to show a small text ad on the side when someone searched for a relevant keyword, I might spend a few dollars here".
Someone submitted the "Strawberry Pop-Tart Blow-Torches" page, which is one of my earliest internet memories. Whoever submitted that, thank you for the nostalgia!
Sounds too good to be true. What's the catch?
The catch is: this is 0.1 alpha software. I need a small team and some server capacity to get rolling. I need people to submit URLs. And a few hundred queries per second. That would scare the living shit out of big league search engines and might wake up some investor wanting to throw money at this.
This project looks neat, I think first experiences with it would be much more improved if you could seed it with some content.
Maybe this could run my search with other search engines to compare and gain insights.
Is this supposed to be a joke? I can't tell. The index is certainly extremely limited.
The challenge is not building the search engine, it’s in building the index. That’s why google wants every drop of data
Is there any further technical documentation than this (besides the source code)?
I tried searching some of the terms in this description on Google, but found little specific information. One search turned up k-d trees. Is this related?