Viewing profile — josefcullhed

Great article, I really think programmers should be more conservative when it comes to new technology. You don't always have to use fancy react stuff when making simple websites an…

Great job with Marginalia. Do you plan to open source your data as well or only the code?

Thanks! The active index is running on 4 servers and we have one server for hosting the frontend and the api (the API is what is used by the frontend, ex: https://api.alexandria.or…

I just think that the timing is right. I think we are in a spot in time where it does not cost billions of dollars to build a search engine like it did 20 years ago. The relevant p…

This is actually not used yet. Working on implementing that as a factor.

We are currently just doing an intersection and then we make a lookup in a forward index to get the urls, titles and snippets. I actually don't know what roaring bitmaps are, pleas…

Founder here. We are using harmonic centrality instead of pagerank. But of course much more work needs to be done to make the search engine usable.

Yes it would be impossible to keep the index in RAM. >> It's hard for me to see how that could be done much faster unless you find a way to parallelize the process We actually para…

Hi, Yes our documentation is probably pretty confusing. It works like this, the base score for all URLs to a specific domain is the harmonic centrality (hc). Then we have two index…

Founder here, I suggest you start by not implementing a crawler but use commoncrawl.org instead. The problem with starting a web crawler is you will need a lot of money and almost …

The index we are running right now are all URLs in commoncrawl from 2021 but only URLs with direct links to them. This is mostly because we would need more servers to index more UR…

Hello, My name is Josef Cullhed. I am the programmer of alexandria.org and one of two founders. We want to build an open source and non profit search engine and right now we are de…

Interesting, in Sweden I only got this story when I made the same searches: https://imgur.com/a/k1Avbtm