Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A search engine that favors text-heavy sites and punishes modern web design

search.marginalia.nu

Post reply on HN
Print view View original

721–730 of 735 posts

Re: A search engine that favors text-heavy sites and punishes modern web design

#721

Post by dahauns »

mananaysiempre wrote: post #632
dahauns wrote: post #583

Earlier quoted context omitted.

English is more the outlier in regard to Germanic languages, try German or Finnish, with their wonderful compounds :) https://e.humanities.uva.nl/publications/2004/kamp_lang04.pd...

Well yeah, English is kind of weird, but Finnish isn’t a Germanic language at all? It’s not even Indo-European, so even Hindi is ostensibly closer to English than Finnish. I understand Standard German (along with Icelandic) is itself a bit atypical in that it hasn’t lost its cases when most other Germanic languages did. Re compounds, I expected they would be more or less easy to deal with by relatively dumb splitting…

Gaaah, brain fart - you're right, of course, dunno why I included it.

Re: A search engine that favors text-heavy sites and punishes modern web design

#722

Post by Aissen »

marginalia_nu wrote: post #720
Aissen wrote: post #717

Earlier quoted context omitted.

I have a static website (granted, it's not well linked), with no JS, and it does not seem to be in the index. But I saw in a sibling post that you had index limits, so it makes sense IMO.

Probably just hasn't been discovered then. The Internet is big. I wish more small websites would be better at linking to other small websites :-(

It must be close, I found a page that links to it. Only one or two more other of magnitudes to index it :-)

Re: A search engine that favors text-heavy sites and punishes modern web design

#723

Post by illys »

titzer wrote: post #582
phendrenad2 wrote: post #549

Earlier quoted context omitted.

Search engines should be something you pay for. Surely search engine powerusers can afford to pay for such a service. If Google makes $1 per user per month or something, that's not too high a bar to get over.

Search engines should be like libraries. At least some tiny sliver of the billions we spend on education and research should go to, you know, actually organizing the world's information and making it universally available.

I see another issue here: companies like Google prioritize information to 1) keep their users and 2) maximize their profit.

If you move data organization to another type of organization (non-profit, state, universities - private or public), then the question of data prioritization becomes highly political. What should be exposed? What should not? What to put first? ...

It is already, but to a smaller extend since money-making companies have little interest in data meaning, and high interest in the commercial value of their users.

Re: A search engine that favors text-heavy sites and punishes modern web design

#724

Post by niutech »

marginalia_nu wrote: post #61
asah wrote: post #46

Earlier quoted context omitted.

Love it, kudos! This is great for developers and others who Just Need Answers and not shopping or entertainment. If you're looking for feedback, both from a UI design and utility standpoint, you might consider "inlining" results from selected sites, e.g. Wikipedia, stacked change, etc. Having worked on search for a long time, inlining (onebox etc) is a big reason users choose Google, and that channelers fail to get t…

If Google starts showing interesting text-heavy links instead of vapid listicles and storefronts, I have accomplished everything I ever could dream of.

You can check the Web Vitals score of Google SERP-s using Core SERP Vitals (https://chrome.google.com/webstore/detail/core-serp-vitals/o...) and filter out the worst results.

Re: A search engine that favors text-heavy sites and punishes modern web design

#725

Post by Valkhyr »

marginalia_nu wrote: post #498
Valkhyr wrote: post #472

As a quick test, I searched for the name of one of my favorite game series: "Baldur's Gate" (on its own, no qualifiers, properly spelled - I would usually spell it "baldurs gate" on Google, but I decided to give this one the best chance). I search for info around video games a lot, so that's quite representative of a good chunk of my web searches, and I pretty much know the top sites Google would give me for that que…

This just isn't the place to go for promotional materials about upcoming video games. It's a niche search engine for discovering stuff off the beaten path, the stuff you can't find on mainstream search engines. Some of it is junk, admittedly, and not everyone will see the point, that's fine too. Despite what some people seem to think, it's never been meant as a google-replacement. I have never claimed otherwise.

Fair enough.

I tried it out under the assumption that it was an attempt at a better mainstream search engine, but I guess I should have paid deeper attention to the name :-)

Re: A search engine that favors text-heavy sites and punishes modern web design

#726

Post by soheil »

marginalia_nu wrote: post #505
Aeolun wrote: post #448

Earlier quoted context omitted.

I’m not sure how you go from word to url range? Range implies contiguous, but how can you make that happen for a bunch of words without keeping track of a list of urls for each word (or URL ids, the idea is the same)?

The trick is that the list of URLs for each word already is in the URLs file. The URLs in a range are sorted. A sorted list (or list-range) forms an implicit set-like data structure, where you can do binary searches to test for existence. Consider a words file with two words, "hello" and "world", corresponding to the ranges (0,3), (3,6). The URLs file contains URLs 1, 5, 7, 2, 5, 8. The first range corresponds to the…

So every time you insert a new URL for a word you have to update the range for every other single word since the URL file will be shifted?

Re: A search engine that favors text-heavy sites and punishes modern web design

#727

Post by ad404b8a372f2b9 »

marginalia_nu wrote: post #712
ad404b8a372f2b9 wrote: post #44

Earlier quoted context omitted.

Very cool project! How many websites do you have in your index? And how did you go about building it? I've been working on an engine for personal websites, currently trying to build a classifier to extract them from commoncrawl, if you have any general tips on that kind of project they'd be very welcome.

About 21 million, and I'm crawling myself. Classification is really hard. I'm struggling with it myself, as a lot of like privacy policies and change logs turns out to share the shape of a page of text. I'm thinking of experimenting with ML classifiers, as I do have reasonably good ways of extracting custom datasets. Finding change logs and privacy policies is easy, excluding them is hard.

If you're open to sharing your index I could make a classifier for you, I do this for a living. It's more of the indexing and search engine part which have been a problem for me. That's why I'm working from commoncrawls.

Re: A search engine that favors text-heavy sites and punishes modern web design

#728

Post by soheil »

marginalia_nu wrote: post #505
Aeolun wrote: post #448

Earlier quoted context omitted.

I’m not sure how you go from word to url range? Range implies contiguous, but how can you make that happen for a bunch of words without keeping track of a list of urls for each word (or URL ids, the idea is the same)?

The trick is that the list of URLs for each word already is in the URLs file. The URLs in a range are sorted. A sorted list (or list-range) forms an implicit set-like data structure, where you can do binary searches to test for existence. Consider a words file with two words, "hello" and "world", corresponding to the ranges (0,3), (3,6). The URLs file contains URLs 1, 5, 7, 2, 5, 8. The first range corresponds to the…

Are the n-grams always at most n=2 bigrams?

Re: A search engine that favors text-heavy sites and punishes modern web design

#729

Post by marginalia_nu »

soheil wrote: post #728
marginalia_nu wrote: post #505

Earlier quoted context omitted.

The trick is that the list of URLs for each word already is in the URLs file. The URLs in a range are sorted. A sorted list (or list-range) forms an implicit set-like data structure, where you can do binary searches to test for existence. Consider a words file with two words, "hello" and "world", corresponding to the ranges (0,3), (3,6). The URLs file contains URLs 1, 5, 7, 2, 5, 8. The first range corresponds to the…

Are the n-grams always at most n=2 bigrams?

No, I actually count the n-grams as distinct words (up to 4-grams). The main limiter is for that is space, so I only extract "canned" n-grams from some tags.

I would first search for the bigram hello_world, that's an O(1) array lookup; as then documents merely containing the words hello and world (usually not a good search result), that's the algorithm I'm describing in the parent comment.

Re: A search engine that favors text-heavy sites and punishes modern web design

#730

Post by soheil »

marginalia_nu wrote: post #729
soheil wrote: post #728

Earlier quoted context omitted.

Are the n-grams always at most n=2 bigrams?

No, I actually count the n-grams as distinct words (up to 4-grams). The main limiter is for that is space, so I only extract "canned" n-grams from some tags. I would first search for the bigram hello_world, that's an O(1) array lookup; as then documents merely containing the words hello and world (usually not a good search result), that's the algorithm I'm describing in the parent comment.

Makes sense. Every time you insert a new URL for a word you have to update the ranges for every other word since the URL file will be shifted?
Post reply on HN