Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dawn of a new era in Search: Balancing innovation, competition, and public good

blog.kagi.com

Post reply on HN
Print view View original

61–70 of 83 posts

Re: Dawn of a new era in Search: Balancing innovation, competition, and public good

#61

Post by 1vuio0pswjnm7 »

andrewstuart wrote: post #44

Who knows what websites and pages are even on the web? There's no index to the web that I know of apart from Google and DuckDuckGo and maybe this Kagi thing. I want to explore the web - surely search isn't the only way to use the web? I imagine it could be fun to explore the web, lists and graphs of interest where I can hop from here to there via list of links or graphs or nodes or something? Does anyone know of anyt…

There have been some websites in the past that allowed one to browse www content by IP address, covering what seemed to be the full range of IPv4 address space. For example, a page with a list of IP address ranges where each address range is a hyperlink. One could then drill down by following hyperlinks to a specific IP address and view whatever was hosted at that address (default host in the case of virtual hosting). Not sure why these websites do not persist. Quite useful. IMHO.

DNS zone files are a decent starting point for exploring the web. Not every registered domain name has an associated website but most do. The largest zone files are available to the public for free.

Re: Dawn of a new era in Search: Balancing innovation, competition, and public good

#62

Post by arnaudsm »

1vuio0pswjnm7 wrote: post #60
arnaudsm wrote: post #48

Earlier quoted context omitted.

I don't buy this number. Text-only common crawl is 20TB. Remove spam and dupes, you're around It's the full Google index history with full HTML that is probably 12PB, but the useful part of the search engine is much smaller.

Does CC publish the methodology for how they determine what to crawl. More particularly, how do they determine what not to crawl.

Yes a few big sites are missing, notably reddit. Most of CC is spam though, the real useful content is really small.

I'm experimenting my own search engine at the moment, and am considering to make it public at some point. It's not that impossible of a task!

Re: Dawn of a new era in Search: Balancing innovation, competition, and public good

#63

Post by Fire-Dragon-DoL »

cube2222 wrote: post #6

I've been using Kagi for a while (almost two years now!) and it's been nothing but excellent! Lenses are very useful (Reddit lens is on every second search), and I personally really like the AI features they are working on. The quick assist triggered by a question mark at the end of a search query which makes a quick ai-generated summary of the few top results is something I use constantly. The new more advanced assi…

I don't have the advanced assistant. Is it only on ultimate?

Re: Dawn of a new era in Search: Balancing innovation, competition, and public good

#65

Post by pbronez »

andrewstuart wrote: post #44

Who knows what websites and pages are even on the web? There's no index to the web that I know of apart from Google and DuckDuckGo and maybe this Kagi thing. I want to explore the web - surely search isn't the only way to use the web? I imagine it could be fun to explore the web, lists and graphs of interest where I can hop from here to there via list of links or graphs or nodes or something? Does anyone know of anyt…

Here’s the best list of search indexes I’ve come across

https://seirdy.one/posts/2021/03/10/search-engines-with-own-...

Re: Dawn of a new era in Search: Balancing innovation, competition, and public good

#66

Post by pbronez »

blackeyeblitzar wrote: post #41

As someone who hasn’t used paid search services, what sort of problems has this solved for other HN users that make it worth it? How does it compare to “AI” based search tools like Perplexity?

For me, Kagi is super fast and provides high quality, customizable results. Web search Just Works. When I use a new computer/browser and accidentally search with Google, it is a viscerally unpleasant surprise.

As for perplexity - I got a free year of Perplexity because I bought a Rabbit R1. I tried it, wasn’t impressed. I use Kagi’s AI assistant all the time. It’s my primary way of getting information from the web. I just type a a free form question into my address bar, append !expert for general questions or !code for technical ones, and seconds later my question is answered and I’m back to work.

Re: Dawn of a new era in Search: Balancing innovation, competition, and public good

#67

Post by AndroidKitKat »

andrewstuart wrote: post #44

Who knows what websites and pages are even on the web? There's no index to the web that I know of apart from Google and DuckDuckGo and maybe this Kagi thing. I want to explore the web - surely search isn't the only way to use the web? I imagine it could be fun to explore the web, lists and graphs of interest where I can hop from here to there via list of links or graphs or nodes or something? Does anyone know of anyt…

While not quite what you're looking for, Kagi has a "Small Web" feed of sites that are semi-curated blogs. [1][2] I don't know how often it is updated, but I like to poke around every now and then see what's going on in people's corners of the internet.

[1] - https://blog.kagi.com/small-web [2] - https://kagi.com/smallweb

Re: Dawn of a new era in Search: Balancing innovation, competition, and public good

#68

Post by cube2222 »

Fire-Dragon-DoL wrote: post #63
cube2222 wrote: post #6

I've been using Kagi for a while (almost two years now!) and it's been nothing but excellent! Lenses are very useful (Reddit lens is on every second search), and I personally really like the AI features they are working on. The quick assist triggered by a question mark at the end of a search query which makes a quick ai-generated summary of the few top results is something I use constantly. The new more advanced assi…

I don't have the advanced assistant. Is it only on ultimate?

Yeah, most AI features are.

Re: Dawn of a new era in Search: Balancing innovation, competition, and public good

#69

Post by Brian_K_White »

ethbr1 wrote: post #23
cube2222 wrote: post #17

Earlier quoted context omitted.

Here's the docs[0]. It's basically a saved partial filter that you can use when searching. So you can e.g. say you only want to search reddit and have that saved as an easily-accessible button. But there's also predefined ones like "Forums" and "Academic", so that's the general idea. The cool thing is you can enable a Lens when using the AI assistant with internet access, so its searches are also constrained to that…

It's essentially search-space limitation, based on what you want to nudge the engine into responding with. I.e. what power users used to do with Google, before it decided to ignore user intent

I don't know how you could ever do kagi small web with google.

That's not a simple hard filter but changes what types of results are more or less preferred.

Re: Dawn of a new era in Search: Balancing innovation, competition, and public good

#70

Post by erlend_sh »

adamcharnock wrote: post #54
freediver wrote: post #38

Earlier quoted context omitted.

> The article estimates the Google Search Index at 12.5PB. I realize there was a mistake with the estimated number (thanks for pointing out, should be closer to 180 PB for raw crawl data). Since this is speculative and also does not account for other data needed to actually rank pages, hardware to do it in under 500ms at a scale of billions of queries per day and thus can be misleading in terms of true effort to do i…

Thank you for the reply! > it is about creating a searchable index of the web scale that has certain quality level that is simply impossible to do anymore for many reasons that would require another article to explain I am both happy to take your word for it, and also very interested to know more. If you were to write that article then I would love to read it.

Aye, such a follow-up article would greatly help bolster the case being made here, which I’m fully on board with.
Post reply on HN