Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alexandria Search

alexandria.org

Post reply on HN
Print view View original

141–150 of 181 posts

Re: Alexandria Search

#141

Post by Seirdy »

danuker wrote: post #138
coverband wrote: post #67

Earlier quoted context omitted.

Could you paste again as text links? Thanks.

Firefox uses a custom format for compressing the SEARCH ENGINES. I am appalled. I had to do a `pip install lz4` then apply and run this change: https://gist.github.com/Tblue/62ff47bef7f894e92ed5?permalink... And it did not work. I hit a brick wall. I completely lost trust in Firefox. I want a browser created by a non-profit. Thank you Google for corrupting everything you touch. I then found this blessed soul: https:/…

Some of these (Andi, nee Lazyweb; You; SwissCows) are Bing proxies. Gnod is a search launcher, not an engine unto itself.

Many more installable engines are available at https://mycroftproject.com as OpenSearch XML plugins, compatible with Firefox and discoverable by Chromium.

Re: Alexandria Search

#142

Post by pmarreck »

josefcullhed wrote: post #95
pmarreck wrote: post #13

This actually makes me want to build my own web crawler and search

Founder here, I suggest you start by not implementing a crawler but use commoncrawl.org instead. The problem with starting a web crawler is you will need a lot of money and almost all big websites are behind cloudflare so you will be blocked pretty quickly. Crawling is a big issue and most of the issues are non-technical.

First off, nice work!

This seems like a reasonable fallback option but it's also a weaker one. By "most of the issues are non-technical", do you mean that you need special permission from someone like cloudflare to get "crawl rights"?

Re: Alexandria Search

#144

Post by hinkley »

daptaq wrote: post #97
outcoldman wrote: post #38

Earlier quoted context omitted.

> I've realized my searching is basically optimized for google Is it just me, or I feel like Google does not provide anymore good results for me. Like every time I search something completely out of my knowledge, like "How to purchase a property in Mexico", it will give me 100+ results of some results with autogenerated content like "10 best places to buy property in Mexico". And the only way to fix that would be to…

Might be an instance of Goodhart's law: https://en.m.wikipedia.org/wiki/Goodhart's_law If all websites try to optimise for SEO, they undermine the assumption that the evaluation of a search engine is the pure consequence of how well a site satisfies a query.

I really think than one, we are going to have to end up with search engines managing a curated list of 'roots', and two, those roots are going to end up consisting substantially of a mix of more 'human' sites and, let's be honest with ourselves, a certain amount of content that is paying for favoritism.

I think it's very possible that we have effectively raised the noise floor so high that there is no signal, but also likely that perverse incentives from trying to profit off of search engines have made them our enemies instead of our friends.

For instance, does Google favor sites that run their own tools on them? I've stopped paying attention but recall hearing mutterings to that effect. If so then running the tools is a protection racket.

For other perverse incentives: if you try to rank sites by how long someone stays on them before backing out, or searching again, then you end up favoring rabbit-hole sites, that either string you along or suck you into a tangent. "Oh, this must have answered their question about keeping bees," no, they're reading gossip about the Queen of England and have forgotten all about beekeeping.

Re: Alexandria Search

#145

Post by jw1224 »

josefcullhed wrote: post #87

Hello, My name is Josef Cullhed. I am the programmer of alexandria.org and one of two founders. We want to build an open source and non profit search engine and right now we are developing in our spare time and are funding the servers ourselves. We are indexing commoncrawl and the search engine is in a really early stage. We would be super happy to find more developers who want to help us.

I searched for a competitive keyword my SaaS business recently reached #1 on Google for. All of our competitors came up, but we were nowhere to be seen (I gave up after page 5).

Does this mean we’re not in Commoncrawl? Or are there any factors you weight much more heavily than Google might?

Re: Alexandria Search

#146

Post by tzot »

51Cards wrote: post #83

So, interesting thing, how when I visit this site for the first time (in Firefox) is the search box showing a drop down with a bunch of my previous searches? I can't tell where they are from but it is all stuff I have searched for in the past. I thought it might be the browser populating a list but that should be based on same domain. So where is it pulling this from? Some of the searche terms are months, perhaps mor…

I can only assume that Firefox associates filled-in data with the name of the input control; in this case, “q”, which is probably typical for a search inputbox.

Re: Alexandria Search

#147

Post by marginalia_nu »

Seirdy wrote: post #126
josefcullhed wrote: post #95

Earlier quoted context omitted.

Founder here, I suggest you start by not implementing a crawler but use commoncrawl.org instead. The problem with starting a web crawler is you will need a lot of money and almost all big websites are behind cloudflare so you will be blocked pretty quickly. Crawling is a big issue and most of the issues are non-technical.

I've heard from other people who run engines (Right Dao, Gigablast) that this is a major problem; Common Crawl does look helpful, but it's not continuously updated. FWIW, Right Dao uses Wikipedia as a starting point for crawling. Kiwix makes pre-indexed dumps of Wikipedia, StackExchange, and other sites available. Some sort of partnership between crawlers could go a long way. Have you considered contributing content…

There seems to be a threshold where you get greylisted by cloudflare. Not sure if it's requests per day or what they're doing. But I've been able to mostly circumvent it by crawling at a modest rate.

Re: Alexandria Search

#148

Post by pabs3 »

Interesting, searching for "Debian", the third result is the rustc package and the eighth is the GitHub repo for the Debian packaging of bino. I wonder how this search engine does its ranking.

Re: Alexandria Search

#149

Post by aghilmort »

qumpis wrote: post #43

Is there a web that pools multiple search engines results?

per other folks on thread, nearly impossible legally due to TOU/TOS or getting around those restrictions with rotating proxies, remote browsing + proxies, etc.

also part of why going with topic filter approach at Breeze -- so if you search all the web, we'll give you option to open others with indie indexes in new tab - say Mojeek or Yandex -- similar to what airline search engines do

if you switch to say code search, you can use ours or redirect to any one of say PublicWWW, Nerdy Data, or Builtwith

pretty much no other legal way to do it on the main

Re: Alexandria Search

#150

Post by aghilmort »

ffhhj wrote: post #55
version_five wrote: post #30

There have been a few search engines out recently. I'm curious how people evaluate them quickly. I've realized my searching is basically optimized for google and the web that has grown up around it. Also, in 1998 I wasn't as aware of what was out there as I am now. It's pretty rare (even if its possible) that I do a search and come across a completely new site that I haven't heard of before, for anything nontrivial.…

> There have been a few search engines out recently I'd like to try them out, could you mention which?

am founder of Breeze, https://breezethat.com/

1. free version is curated topic filters that sit on top of Google -- best on laptop or desktop at moment / iterating mobile due to ad splash; it's unclear due to TOU that we can ever make that fully server-side legally, though do so have some things testing to see if free version can be ad-free or more tracking free

2. premium version will be mix of our scraping and Bing, depending on topic - standard web search via Bing + same curation as (1), closer to real-time for us

3. have tested out most other indie indexes or sites, for full web scale, money is on ahrefs or brave giving bing / google run for it

4. our primary emphasis is on bringing back some of the Yahoo! directory or Alta Vista look & feel of drilling into topics, so balancing 1-3 (^) as best we can atm, small team, fully bootstrapped modulo tiny F&F round

5. also @DotDotJames on twitter, still iterating when add team info to site

Post reply on HN