Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How would you build a search engine in 2019?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 20 posts

Re: Ask HN: How would you build a search engine in 2019?

#11

Post by ian0 »

ian0 wrote: post #10

In my mind there are two flavours of queries. The first are those which are "ctrl f" in nature, ie I want to query the entire web to find string x. Which are obviously better suited to indexes. The second are more knowledge based in nature. IE I want to find quality information about a certain topic. These benefit from curation, as anyone who has added "stackoverflow" or "reddit" or "wiki" to a google query will know…

Another interesting one is illustrating "depth" of knowledge in a results page. For example, I type "Sagittarius A*". It would be nice to have the results grouped into depth. From initial general-audience explainer videos on youtube down to arvix results. With current news seperated from general informational content.

This works for a lot of niches. For example motorcycle reviews run from entertaining comparisons down to detailed analysis. Recipies for omelettes run from the basics, through adding interesting ingredients, right through to the art of perfectly preparing a difficult to make french omelette.

Re: Ask HN: How would you build a search engine in 2019?

#12

Post by techsin101 »

gfs wrote: post #7
rootshelled wrote: post #4

Earlier quoted context omitted.

Wouldn't blockchain make more sense? (Never thought I would say this but here we are) Problem with torrents/blockchain in this case are twofold: - unless there is enough people using it, it's slow - since you are basically crowdsourcing you need to create an enticing incentive which works for everyone involved

Could you elaborate on how blockchain would make more sense? I am failing to see how it does.

Block chain doesn't

Re: Ask HN: How would you build a search engine in 2019?

#13

Post by ai_ia »

bwb wrote: post #9

I think the why changes the approach... I think my takeaway from the last 10 years, is that a lot of the info on websites that was by real people has disappeared and you have a lot of spammy blog and heavily commercial approaches. And, most of the real info has gone into facebook groups, quora, reddit comments, slack, twitter, and so on. The problem is those are all closed-door eco-systems in a lot of ways, and the k…

>I think if I was going to approach this I would build software that users run, or browser add ons that lets user tag and save information in some type of format, and then that contributes to a knowledge search information.

Something like delicious or pinboard, I suppose.

I agree. A social bookmarking website would be pretty interesting if done right.

Re: Ask HN: How would you build a search engine in 2019?

#14

Post by bwb »

ai_ia wrote: post #13
bwb wrote: post #9

I think the why changes the approach... I think my takeaway from the last 10 years, is that a lot of the info on websites that was by real people has disappeared and you have a lot of spammy blog and heavily commercial approaches. And, most of the real info has gone into facebook groups, quora, reddit comments, slack, twitter, and so on. The problem is those are all closed-door eco-systems in a lot of ways, and the k…

>I think if I was going to approach this I would build software that users run, or browser add ons that lets user tag and save information in some type of format, and then that contributes to a knowledge search information. Something like delicious or pinboard, I suppose. I agree. A social bookmarking website would be pretty interesting if done right.

Well personal + fed into a centralized system, a human knowledge search engine which I think was some of the magic of the early internet but now is buried in FB groups and the other locations i mentioned.

Re: Ask HN: How would you build a search engine in 2019?

#15

Post by fortran77 »

rootshelled wrote: post #4
hakejpc wrote: post #2

Can we leverage torrents somehow to create distributed indices ?

Wouldn't blockchain make more sense? (Never thought I would say this but here we are) Problem with torrents/blockchain in this case are twofold: - unless there is enough people using it, it's slow - since you are basically crowdsourcing you need to create an enticing incentive which works for everyone involved

Torrents + Blockchain + _Rust_ would be even better!

Re: Ask HN: How would you build a search engine in 2019?

#16

Post by K0SM0S »

bwb wrote: post #14
ai_ia wrote: post #13

Earlier quoted context omitted.

>I think if I was going to approach this I would build software that users run, or browser add ons that lets user tag and save information in some type of format, and then that contributes to a knowledge search information. Something like delicious or pinboard, I suppose. I agree. A social bookmarking website would be pretty interesting if done right.

Well personal + fed into a centralized system, a human knowledge search engine which I think was some of the magic of the early internet but now is buried in FB groups and the other locations i mentioned.

Cue some deep learning over this centralized dataset... we might just see 'profiles' emerge that allow the system to build very ad hoc results depending on who searches what.

E.g. one of the strengths of Google (which I've used for a decade without ever deleting history) is to give me very hands-on blogs and procedures whenever I search for some tech — I don't even have to add "guide" or "tutorial" or "hands-on" or "how to", whenever I use my dev account Google just knows what I'm after — “this guy MAKES, he don't care for fluff”. I've compared searches with friends (non-tech non-nerds), they get the usual wiki / reddit / commercial stuff, with my meaty top 10 stuff often on page 2-3-4 (if even that!) for them.

This is the kind of smarts I don't know how to replicate without ML. This is also where Spotify who has had years to figure it out remains extremely generic (frankly bad at deep cuts, which at a basic level used to be one of their initial strengths).

Re: Ask HN: How would you build a search engine in 2019?

#17

Post by siquick »

bwb wrote: post #9

I think the why changes the approach... I think my takeaway from the last 10 years, is that a lot of the info on websites that was by real people has disappeared and you have a lot of spammy blog and heavily commercial approaches. And, most of the real info has gone into facebook groups, quora, reddit comments, slack, twitter, and so on. The problem is those are all closed-door eco-systems in a lot of ways, and the k…

Fully agreed with this. In the early to mid 00's I was a heavy user of electronic music forums, everyday I was learning about new artists/releases/labels and there were hundreds of active users discussing new music on most of the boards.

Last night I spent a good hour or so trying to find something similar and just couldn't find anything. The closest thing now is Reddit but the quality of discussion just isn't what it was during those golden phpbb forum days. It's seems like it's either Facebook groups or nothing.

Re: Ask HN: How would you build a search engine in 2019?

#18

Post by probinso »

I would build a discovery system. Search systems imply that you know what you're looking for, it also implies a short query.

Imagine instead you start writing out your ideas in natural form. Documents will appear that are relevant to your ideas, but with the goal of diversifying category. Instead of the top 100 results, you may just get minimal results per category.

As you continue to write the relevant documents get more constraint, but continue to attempt to maximize diversity.

I don't know if this work for everything, but would be interesting.

Re: Ask HN: How would you build a search engine in 2019?

#19

Post by quickthrower2 »

fortran77 wrote: post #15
rootshelled wrote: post #4

Earlier quoted context omitted.

Wouldn't blockchain make more sense? (Never thought I would say this but here we are) Problem with torrents/blockchain in this case are twofold: - unless there is enough people using it, it's slow - since you are basically crowdsourcing you need to create an enticing incentive which works for everyone involved

Torrents + Blockchain + _Rust_ would be even better!

Don’t forget webassembly

Re: Ask HN: How would you build a search engine in 2019?

#20

Post by kamutuna »

with something like proxycrawl you should be able to build the crawler yourself and start crawling to build a big index. Once you get big enough and you start bringing traffic to the sites, you can then stop using them and create your own user agent and ask sites to whitelist you. It will take some time but can be done
Post reply on HN