Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alexandria Search

alexandria.org

Post reply on HN
Print view View original

101–110 of 181 posts

Re: Alexandria Search

#101

Post by demopathos »

version_five wrote: post #30

There have been a few search engines out recently. I'm curious how people evaluate them quickly. I've realized my searching is basically optimized for google and the web that has grown up around it. Also, in 1998 I wasn't as aware of what was out there as I am now. It's pretty rare (even if its possible) that I do a search and come across a completely new site that I haven't heard of before, for anything nontrivial.…

I'm surprised you are not a fan of geeksforgeeks. While each of their webpages have substantially less content than the pytorch docs or SO result, I find that they get to the point instantly. My mean time to solution from G4G is definitely smaller than SO.

Re: Alexandria Search

#102

Post by RosanaAnaDana »

demopathos wrote: post #101
version_five wrote: post #30

There have been a few search engines out recently. I'm curious how people evaluate them quickly. I've realized my searching is basically optimized for google and the web that has grown up around it. Also, in 1998 I wasn't as aware of what was out there as I am now. It's pretty rare (even if its possible) that I do a search and come across a completely new site that I haven't heard of before, for anything nontrivial.…

I'm surprised you are not a fan of geeksforgeeks. While each of their webpages have substantially less content than the pytorch docs or SO result, I find that they get to the point instantly. My mean time to solution from G4G is definitely smaller than SO.

Geeksforgeeks is toxic garbage poisoning the well of good solutions for common problems.

Re: Alexandria Search

#103

Post by josefcullhed »

cocoafleck wrote: post #96
josefcullhed wrote: post #87

Hello, My name is Josef Cullhed. I am the programmer of alexandria.org and one of two founders. We want to build an open source and non profit search engine and right now we are developing in our spare time and are funding the servers ourselves. We are indexing commoncrawl and the search engine is in a really early stage. We would be super happy to find more developers who want to help us.

I was trying to learn more about the ranking algorithm that Alexandria uses, and I was a bit confused by the documentation on Github for it. Would I be correct in that it uses "Harmonic Centrality" ( http://vigna.di.unimi.it/ftp/papers/AxiomsForCentrality.pdf ) at least for part of the algorithm?

Hi,

Yes our documentation is probably pretty confusing. It works like this, the base score for all URLs to a specific domain is the harmonic centrality (hc). Then we have two indexes, one with URLs and one with links (we index the link text). Then we first make a search on the links, then on the URLs. We then update the score of the urls based on the links with this formula: domain_score = expm1(5 * link.m_score) + 0.1; url_score = expm1(10 * link.m_score) + 0.1;

then we add the domain and url score to url.m_score

where link.m_score is the HC of the source domain.

Re: Alexandria Search

#104

Post by josefcullhed »

kreeben wrote: post #100
josefcullhed wrote: post #92

Earlier quoted context omitted.

The index we are running right now are all URLs in commoncrawl from 2021 but only URLs with direct links to them. This is mostly because we would need more servers to index more URLs and that would increase the cost. It takes us a couple of days to build the index but we have been coding this for about 1 year. All the indexes are on disk.

>> All the indexes are on disk. Love it. Makes for a cheaper infrastructure, since SSD is cheaper than RAM. >> It takes us a couple of days to build the index It's hard for me to see how that could be done much faster unless you find a way to parallelize the process, which in itself is a terrifyingly hard problem. I haven't read your code yet, obviously, but could you give us a hint as to what kind of data structure…

Yes it would be impossible to keep the index in RAM.

>> It's hard for me to see how that could be done much faster unless you find a way to parallelize the process

We actually parallelize the process. We do it by separating the URLs to three different servers and indexing them separately. Then we just make the searches on all three servers and merges the result URLs.

>> I haven't read your code yet, obviously, but could you give us a hint as to what kind of data structure you use for indexing?

It is not very complicated, we use hashes a lot to simplify things. The index is basically a really large hash table with the word_hash -> [list of url hashes] Then if you search for "The lazy fox" we just take the intersection between the three lists of url hashes to get all the urls which have all words in them. This is the basic idea that is implemented right now but we will of course try to improve.

details are here: https://github.com/alexandria-org/alexandria/blob/main/src/i...

Re: Alexandria Search

#105

Post by _bohm »

demopathos wrote: post #101
version_five wrote: post #30

There have been a few search engines out recently. I'm curious how people evaluate them quickly. I've realized my searching is basically optimized for google and the web that has grown up around it. Also, in 1998 I wasn't as aware of what was out there as I am now. It's pretty rare (even if its possible) that I do a search and come across a completely new site that I haven't heard of before, for anything nontrivial.…

I'm surprised you are not a fan of geeksforgeeks. While each of their webpages have substantially less content than the pytorch docs or SO result, I find that they get to the point instantly. My mean time to solution from G4G is definitely smaller than SO.

I've found that their content is often inaccurate or written by people who come across as novices. I actually emailed them to correct an inaccuracy in one of their articles once, which they did, so kudos to them for that.

Re: Alexandria Search

#106

Post by josefcullhed »

waynecochran wrote: post #94
stazz1 wrote: post #3

Earlier quoted context omitted.

>About Alexandria Alexandria.org is a non-profit, ad free search engine. Our goal is to provide the best available information without compromise. The index is built on data from Common Crawl and the engine is written in C++. The source code is available here. We are still at an early stage of development and running the search engine on a shoestring budget. Please contact us at -email- if you want to get involved, w…

But what is different in terms of its indexing algorithm? The original secret sauce for google was the pagerank algorithm which was mathematically genius. Are you using a similar algorithm.

Founder here. We are using harmonic centrality instead of pagerank. But of course much more work needs to be done to make the search engine usable.

Re: Alexandria Search

#107

Post by kreeben »

josefcullhed wrote: post #104
kreeben wrote: post #100

Earlier quoted context omitted.

>> All the indexes are on disk. Love it. Makes for a cheaper infrastructure, since SSD is cheaper than RAM. >> It takes us a couple of days to build the index It's hard for me to see how that could be done much faster unless you find a way to parallelize the process, which in itself is a terrifyingly hard problem. I haven't read your code yet, obviously, but could you give us a hint as to what kind of data structure…

Yes it would be impossible to keep the index in RAM. >> It's hard for me to see how that could be done much faster unless you find a way to parallelize the process We actually parallelize the process. We do it by separating the URLs to three different servers and indexing them separately. Then we just make the searches on all three servers and merges the result URLs. >> I haven't read your code yet, obviously, but co…

I realize I'm asking for a free ride here, but could you explain what happens after the index scan? In a phrase search you'd need to intersect, union or remove from the results. Are you using roaring bitmaps or something similar?

Re: Alexandria Search

#108

Post by randomsilence »

zeta0134 wrote: post #68
version_five wrote: post #30

There have been a few search engines out recently. I'm curious how people evaluate them quickly. I've realized my searching is basically optimized for google and the web that has grown up around it. Also, in 1998 I wasn't as aware of what was out there as I am now. It's pretty rare (even if its possible) that I do a search and come across a completely new site that I haven't heard of before, for anything nontrivial.…

I've been wondering for a while now about building a search engine for the ad free web. That is, penalize or outright refuse to index any recognized advertising network, letting through only those sites which don't perform invasive tracking with third party services. Mostly as a curiosity: what would be left? What would rise to the top when you filter all of that out?

Check the 'Small or non-commercial Web' search engines on this overview page: https://seirdy.one/2021/03/10/search-engines-with-own-indexe...

Re: Alexandria Search

#109

Post by josefcullhed »

josefcullhed wrote: post #104
kreeben wrote: post #100

Earlier quoted context omitted.

>> All the indexes are on disk. Love it. Makes for a cheaper infrastructure, since SSD is cheaper than RAM. >> It takes us a couple of days to build the index It's hard for me to see how that could be done much faster unless you find a way to parallelize the process, which in itself is a terrifyingly hard problem. I haven't read your code yet, obviously, but could you give us a hint as to what kind of data structure…

Yes it would be impossible to keep the index in RAM. >> It's hard for me to see how that could be done much faster unless you find a way to parallelize the process We actually parallelize the process. We do it by separating the URLs to three different servers and indexing them separately. Then we just make the searches on all three servers and merges the result URLs. >> I haven't read your code yet, obviously, but co…

We are currently just doing an intersection and then we make a lookup in a forward index to get the urls, titles and snippets.

I actually don't know what roaring bitmaps are, please enlighten me :)

Re: Alexandria Search

#110

Post by marginalia_nu »

lukasb wrote: post #99
marginalia_nu wrote: post #76

Earlier quoted context omitted.

That's easy to do (it's Personalized PageRank), but VERY expensive. Like just tossing them a few dollars doesn't cut it. You basically need your own custom index for that, as the way you achieve fast ranking is by sorting the documents in order of ranking within the index itself. That way you only need to consider a very small portion of the index to retrieve the highest ranking results. You might get away with havin…

So uh ... 100 petabytes, then. Ah.

Realistically you could probably get away with something like a couple of terabytes, and then default to the regular index if it isn't found in the neighborhood close to your favored sites, but that's still anything but cheap especially this can't be some slow-ass S3 storage, this storage should ideally be SSDs or a RAID/JBOD-configuration of mechanical drives. That means you're also paying for a lot of I/O bandwidth and overall data logistics.

If you try to rent that sort of compute, you're probably looking at like $100-200/month.

Post reply on HN