701–710 of 735 posts
This is stunning. I searched "winemaking" because it's my latest obsession, and turned up dozens of links to high-quality pages I'd never seen despite spending an hour a day for three months cruising Google on the topic. Please do announce it here if you ever decide to solicit help or contributors. My stab at this problem was to have a search index of only ad-free pages, on the hypothesis it would turn up self-hosted…
Earlier quoted context omitted.
The trick is that the list of URLs for each word already is in the URLs file. The URLs in a range are sorted. A sorted list (or list-range) forms an implicit set-like data structure, where you can do binary searches to test for existence. Consider a words file with two words, "hello" and "world", corresponding to the ranges (0,3), (3,6). The URLs file contains URLs 1, 5, 7, 2, 5, 8. The first range corresponds to the…
Hm, ok I understand more but how do you perform the "binary search", just loop over the URL ids? Funny I also selected "hello" and "world" above! Xo My system is also written in Java btw! Here are example results of my word search: http://root.rupy.se/node/data/word/four http://root.rupy.se/node/data/word/only etc.
But yeah, in short pseudocode:
for url in range-for-"hello":
if binary-search (range-for-"world", url):
yield url
Earlier quoted context omitted.
What crawler are you using and what kind of crawling speeds are you achieving? How do you rank the results (is it based on content only) or you have external factors too? What is your personal preferred search option of the 7 and why? Thanks for making something unique and sorry that despite all the hype this got, you got only $39/month on Patreon. It is telling in a way.
> What crawler are you using and what kind of crawling speeds are you achieving? Custom crawler, and I seem to get around 100 documents per second at best, maybe closer to 50 on average. Depends a bit on how many crawl-worthy websites it finds, and there is definitely diminishing returns as it goes deeper. >How do you rank the results (is it based on content only) or you have external factors too? I rank based on a p…
And I am not kidding. I think for something that got so much attention on HN, where realistically this kind of product can only exist for now, the 'conversion' rate was very low. Billion dollar companies were made of HN threads with lot less engagement. Makes me wonder do we really want a search engine like this or we just like the idea of it?
And what are the barriers to use something like this? You say yourself that you are using Google most of the time. Is jumping to check results on this engine going to be too much friction for most uses?
Can something like this exist in isolation? What kind of value would it need to provide for users to remember using it en-masse as an additional/primary vertical search like they do for Amazon?
Just thinking out-loud as I am also interested in the space (through http://teclis.com).
Earlier quoted context omitted.
> What crawler are you using and what kind of crawling speeds are you achieving? Custom crawler, and I seem to get around 100 documents per second at best, maybe closer to 50 on average. Depends a bit on how many crawl-worthy websites it finds, and there is definitely diminishing returns as it goes deeper. >How do you rank the results (is it based on content only) or you have external factors too? I rank based on a p…
You have a great attitude! And I am not kidding. I think for something that got so much attention on HN, where realistically this kind of product can only exist for now, the 'conversion' rate was very low. Billion dollar companies were made of HN threads with lot less engagement. Makes me wonder do we really want a search engine like this or we just like the idea of it? And what are the barriers to use something like…
Ultimately I think running something like this for profit would create really unhealthy incentives to make my search engine worse. Any value it brings, right now, it brings because it isn't trying to cater to every taste and every use case.
I also hate the constant "don't forget to slap the like and subscribe buttons"-shout outs of modern social media, even though I'm aware they it is extremely efficient. If I went down that route, I would become part of the problem I'm trying to help cure. I do feel the sirens' call though, it's intoxicating getting this sort of praise and attention.
I want this to be a long-term project, not a some overnight cinderella story.
In the end, my search engine is never going to replace google. It isn't trying to, it's trying to complement it. It's barely able now, but hopefully I can make it much better in the months and years to come.
Earlier quoted context omitted.
What crawler are you using and what kind of crawling speeds are you achieving? How do you rank the results (is it based on content only) or you have external factors too? What is your personal preferred search option of the 7 and why? Thanks for making something unique and sorry that despite all the hype this got, you got only $39/month on Patreon. It is telling in a way.
> What crawler are you using and what kind of crawling speeds are you achieving? Custom crawler, and I seem to get around 100 documents per second at best, maybe closer to 50 on average. Depends a bit on how many crawl-worthy websites it finds, and there is definitely diminishing returns as it goes deeper. >How do you rank the results (is it based on content only) or you have external factors too? I rank based on a p…
Plus another excellent feature was you would get the same search results no matter who or where you were for quite some period of calendar time.
If something new did appear it was likely to be one of the new sites that was popping up all the time and it was likely to be as worthwhile as its established associates on the front page.
You shouldn't need to crawl nearly as fast if you can compensate by treading more suitably where those have gone before.
Earlier quoted context omitted.
You have a great attitude! And I am not kidding. I think for something that got so much attention on HN, where realistically this kind of product can only exist for now, the 'conversion' rate was very low. Billion dollar companies were made of HN threads with lot less engagement. Makes me wonder do we really want a search engine like this or we just like the idea of it? And what are the barriers to use something like…
I think in part it may just be because I'm not trying to found a start-up, and I'm not trying to get rich quick. If I were, I would have dealt with this very differently. My integrity and the integrity of this project is far more important than my bank balance. Not everyone feels that way, and I can respect that, but I do. Ultimately I think running something like this for profit would create really unhealthy incenti…
So given your setup is already ideal for 'conversions' for this population (low profile, high integrity, no BS) I was simply genuinely surprised that only 11 people converted given enormous visibility/interest this thread had. Hope that makes sense.
Earlier quoted context omitted.
You have a great attitude! And I am not kidding. I think for something that got so much attention on HN, where realistically this kind of product can only exist for now, the 'conversion' rate was very low. Billion dollar companies were made of HN threads with lot less engagement. Makes me wonder do we really want a search engine like this or we just like the idea of it? And what are the barriers to use something like…
I think in part it may just be because I'm not trying to found a start-up, and I'm not trying to get rich quick. If I were, I would have dealt with this very differently. My integrity and the integrity of this project is far more important than my bank balance. Not everyone feels that way, and I can respect that, but I do. Ultimately I think running something like this for profit would create really unhealthy incenti…
This allows quite a bit of its own kind of freedom even if maximum financial opportunity is not fully exploited. Perhaps even because you are not grasping for every dollar on the table at all times.
You can do things without having to know if they will pay off, and if it turns out big anyway you can make money as a byproduct of what you do rather than having pure financial pursuit be the root of every goal.
Earlier quoted context omitted.
This isn't meant to be pressurizing or to sound like a demand (if it seems that way), but have you thought about uploading the source code for your search engine? Something like this has the potential to be used in university courses to teach how to build a search engine and/or teach 'advanced' programming concepts and ideas. It's a real program showing what you need to do to optimize your database and software to wo…
I'm thinking I may open source some of the components I use, rather than publishing the whole thing, as it's part of a larger monorepo that contains the somewhat integrated code for a large set of services, public and private. None of the code is particularly fancy, just highly specialized. I did build them myself mostly because I couldn't find anything available that the rather special demands that are put on the ap…
Also, if I'm not being too rude, your posts were a little hard for me to understand, so I thought that releasing some or all of the code would help people (and myself) understand them better, ignoring the teaching stuff that I already talked about. With code, people can debug and change it to show what the program's doing. It'd also at least satisfy some people's curiosity (and probably use up their weekend).
And thanks for replying!
super! How far would you say are you in indexing the blogosphere ? I tried the engine a few times, but I mostly get academic papers and I know most (good) blogs are in fact text-heavy.
Earlier quoted context omitted.
I think in part it may just be because I'm not trying to found a start-up, and I'm not trying to get rich quick. If I were, I would have dealt with this very differently. My integrity and the integrity of this project is far more important than my bank balance. Not everyone feels that way, and I can respect that, but I do. Ultimately I think running something like this for profit would create really unhealthy incenti…
I agree with everything you say. The 'subscribe and like buttons' would not help your conversion with HN readers, on the contrary. Trying to run this for profit also would not help your conversion with this audience. So given your setup is already ideal for 'conversions' for this population (low profile, high integrity, no BS) I was simply genuinely surprised that only 11 people converted given enormous visibility/in…
I'd absolutely consider sending someone money if they kept bringing something of value into my life. If I want more people to join the patreon, I'll just have to earn their trust and support.