Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alexandria Search

alexandria.org

Post reply on HN
Print view View original

91–100 of 181 posts

Re: Alexandria Search

#91

Post by jll29 »

amelius wrote: post #15
tandr wrote: post #9

I think the fact that after a long while there are new search engines (Kagi was introduced very recently on HN, now this) should be a wake up call for Google - their search has lost some shine for quite a while. Hopefully something will come out of this - competition is good.

At some point, AI and NLP and raw processing power will have progressed so much that "search" is not a problem anymore, and I think we're getting there. Google can up their game but it won't matter much. The only thing they have left is brand recognition.

Google Research is also one of the top (NLR|IR) R&D gigs in town - they discovered BERT, a model that has re-defined how NLP is down and the respective paper describing it already collected 800 citations by the time it was published based on a pre-print spreading like a wildfire.

This technology is now part of Google search.

Re: Alexandria Search

#92

Post by josefcullhed »

kreeben wrote: post #90
josefcullhed wrote: post #87

Hello, My name is Josef Cullhed. I am the programmer of alexandria.org and one of two founders. We want to build an open source and non profit search engine and right now we are developing in our spare time and are funding the servers ourselves. We are indexing commoncrawl and the search engine is in a really early stage. We would be super happy to find more developers who want to help us.

Thanks for sharing this with the world. Did you manage to include all of a common crawl in an index? How long did that take you to produce such an index? Is your index in-memory or on disk? I'd consider contributing. Seems you have something here.

The index we are running right now are all URLs in commoncrawl from 2021 but only URLs with direct links to them. This is mostly because we would need more servers to index more URLs and that would increase the cost.

It takes us a couple of days to build the index but we have been coding this for about 1 year.

All the indexes are on disk.

Re: Alexandria Search

#93

Post by marginalia_nu »

kreeben wrote: post #89
marginalia_nu wrote: post #88

Earlier quoted context omitted.

https://en.m.wikipedia.org/wiki/Precision_and_recall For some standard corpus.

>> Precision and recall The phrase I was looking for. Thx a bunch! Gonna marginalia that now.

Haha, ironically it lacks both precision and recall for that topic.

Re: Alexandria Search

#94

Post by waynecochran »

stazz1 wrote: post #3
waynecochran wrote: post #2

In a nutshell, what is the fundamental difference with this search engine compared to others?

>About Alexandria Alexandria.org is a non-profit, ad free search engine. Our goal is to provide the best available information without compromise. The index is built on data from Common Crawl and the engine is written in C++. The source code is available here. We are still at an early stage of development and running the search engine on a shoestring budget. Please contact us at -email- if you want to get involved, w…

But what is different in terms of its indexing algorithm? The original secret sauce for google was the pagerank algorithm which was mathematically genius. Are you using a similar algorithm.

Re: Alexandria Search

#95

Post by josefcullhed »

pmarreck wrote: post #13

This actually makes me want to build my own web crawler and search

Founder here,

I suggest you start by not implementing a crawler but use commoncrawl.org instead. The problem with starting a web crawler is you will need a lot of money and almost all big websites are behind cloudflare so you will be blocked pretty quickly. Crawling is a big issue and most of the issues are non-technical.

Re: Alexandria Search

#96

Post by cocoafleck »

josefcullhed wrote: post #87

Hello, My name is Josef Cullhed. I am the programmer of alexandria.org and one of two founders. We want to build an open source and non profit search engine and right now we are developing in our spare time and are funding the servers ourselves. We are indexing commoncrawl and the search engine is in a really early stage. We would be super happy to find more developers who want to help us.

I was trying to learn more about the ranking algorithm that Alexandria uses, and I was a bit confused by the documentation on Github for it. Would I be correct in that it uses "Harmonic Centrality" (http://vigna.di.unimi.it/ftp/papers/AxiomsForCentrality.pdf) at least for part of the algorithm?

Re: Alexandria Search

#97

Post by daptaq »

outcoldman wrote: post #38
version_five wrote: post #30

There have been a few search engines out recently. I'm curious how people evaluate them quickly. I've realized my searching is basically optimized for google and the web that has grown up around it. Also, in 1998 I wasn't as aware of what was out there as I am now. It's pretty rare (even if its possible) that I do a search and come across a completely new site that I haven't heard of before, for anything nontrivial.…

> I've realized my searching is basically optimized for google Is it just me, or I feel like Google does not provide anymore good results for me. Like every time I search something completely out of my knowledge, like "How to purchase a property in Mexico", it will give me 100+ results of some results with autogenerated content like "10 best places to buy property in Mexico". And the only way to fix that would be to…

Might be an instance of Goodhart's law: https://en.m.wikipedia.org/wiki/Goodhart's_law

If all websites try to optimise for SEO, they undermine the assumption that the evaluation of a search engine is the pure consequence of how well a site satisfies a query.

Re: Alexandria Search

#98

Post by jll29 »

amelius wrote: post #72
version_five wrote: post #30

There have been a few search engines out recently. I'm curious how people evaluate them quickly. I've realized my searching is basically optimized for google and the web that has grown up around it. Also, in 1998 I wasn't as aware of what was out there as I am now. It's pretty rare (even if its possible) that I do a search and come across a completely new site that I haven't heard of before, for anything nontrivial.…

> I'm curious how people evaluate them quickly. Are there search benchmarks to be found somewhere? There must be. If you want to write a search engine, you need a way to validate the results.

The "Web track" task at the annual US NIST TREC conference ( https://trec.nist.gov/ ) is an open innovation benchmark that everyone can contribute; participants get a set of queries that they have to run on exactly the same corpus. Then they return the top-k results to a team that evaluates them.

Here is an example (2014) Web track paper of the 23th TREC: https://trec.nist.gov/pubs/trec23/papers/overview-web.pdf (TREC has a plentitude of difference benchmark tasks and you can submit your own: https://trec.nist.gov/pubs/trec29/trec2020.html - recent TREC 2020 papers)

Re: Alexandria Search

#99

Post by lukasb »

marginalia_nu wrote: post #76
lukasb wrote: post #65

Excuse me while I get on a hobbyhorse - would love to use web search that lets me boost PageRank for certain sites (which then would carry over to sites they link to.) Could automatically boost PageRank for sites I subscribe to, for example. Expensive in terms of computation or storage? Charge me!

That's easy to do (it's Personalized PageRank), but VERY expensive. Like just tossing them a few dollars doesn't cut it. You basically need your own custom index for that, as the way you achieve fast ranking is by sorting the documents in order of ranking within the index itself. That way you only need to consider a very small portion of the index to retrieve the highest ranking results. You might get away with havin…

So uh ... 100 petabytes, then.

Ah.

Re: Alexandria Search

#100

Post by kreeben »

josefcullhed wrote: post #92
kreeben wrote: post #90

Earlier quoted context omitted.

Thanks for sharing this with the world. Did you manage to include all of a common crawl in an index? How long did that take you to produce such an index? Is your index in-memory or on disk? I'd consider contributing. Seems you have something here.

The index we are running right now are all URLs in commoncrawl from 2021 but only URLs with direct links to them. This is mostly because we would need more servers to index more URLs and that would increase the cost. It takes us a couple of days to build the index but we have been coding this for about 1 year. All the indexes are on disk.

>> All the indexes are on disk.

Love it. Makes for a cheaper infrastructure, since SSD is cheaper than RAM.

>> It takes us a couple of days to build the index

It's hard for me to see how that could be done much faster unless you find a way to parallelize the process, which in itself is a terrifyingly hard problem.

I haven't read your code yet, obviously, but could you give us a hint as to what kind of data structure you use for indexing? According to you, what kind of data structure allows for the fastest indexing and how do you represent it on disk so that you can read your on-disk index in a forward-only mode or "as fast as possible"?

Post reply on HN