Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Waiting for dawn in search: Search index, Google rulings and impact on Kagi

blog.kagi.com

Post reply on HN
Print view View original

261–266 of 266 posts

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#261

Post by ricardo81 »

hattmall wrote: post #156
creato wrote: post #145

Earlier quoted context omitted.

The point is, robots.txt was definitely a thing that people expected to be respected before and during google's early existence. This Kagi claim seems to be at least partially false: > Google built its index by crawling the open web before robots.txt was a widespread norm, often over publishers’ objections.

Perhaps it wasn't a widespread norm though. But I don't really see why that matters as much, is the the issue that sites with robots.txt today only allow Googlebot and not other search engines? Or is Google somehow benefitting from having two decade old content that is now blocked because of robots.txt that the website operators don't want indexed?

Agree. It was not standard in the late 90s or early 00s. Most sites were custom built and relied on the _webmaster_ knowing and understanding how robots.txt worked. I'd heard plenty of examples where people had inadvertently blocked crawlers from their site, not knowing the syntax correctly. CMS' probably helped in the widespread adoption e.g. wordpress

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#262

Post by ricardo81 »

dylan604 wrote: post #112
baggachipz wrote: post #83

Earlier quoted context omitted.

A classic case of climbing the wall, and pulling the ladder up afterward. Others try to build their own ladder, and Google uses their deep pockets and political influence to knock the ladder over before it reaches the top.

Why does Google even need to know about your ladder? Build the bot, scale it up, save all the data, then release. You can now remove the ladder and obey robots.txt just like G. Just like G, once you have the data, you have the data. Why would you tell G that you are doing something? Why tell a competitor your plans at all? Just launch your product when the product is ready. I know that's anathema to SV startup logic,…

There's one great example of a company that did that and managed to go viral on their release, Cuil. They claimed to have a Google size of search index. Unfortunately for them their search results weren't good and so that visibility quickly disappeared.

Going further back, AlltheWeb was actually pretty decent but was eventually bought by Overture and then Yahoo and ended up in their graveyard.

For everyone else it's the longer grind trying to gain visibility.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#263

Post by b3kart »

charcircuit wrote: post #108
b3kart wrote: post #78

Earlier quoted context omitted.

I think you’re proving the monopoly argument yourself: if they only way to compete with Google is an innovation that generations of scientists have been working towards, it does paint a grim picture of competition in this space. Besides, are we ignoring Gemini?

Google already used AI and language models before ChatGPT came out. If you wanted a state of the art search / recommendation engine you needed that innovations from scientists already.

That's what I am saying: if you had a better search/rec engine than Google, good luck making it useful without Google's search index, acquired to a large extent thanks to their dominant market position. This doesn't sound like healthy competition. ChatGPT had to change the whole game to be able to compete.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#264

Post by baggachipz »

ricardo81 wrote: post #262
dylan604 wrote: post #112

Earlier quoted context omitted.

Why does Google even need to know about your ladder? Build the bot, scale it up, save all the data, then release. You can now remove the ladder and obey robots.txt just like G. Just like G, once you have the data, you have the data. Why would you tell G that you are doing something? Why tell a competitor your plans at all? Just launch your product when the product is ready. I know that's anathema to SV startup logic,…

There's one great example of a company that did that and managed to go viral on their release, Cuil. They claimed to have a Google size of search index. Unfortunately for them their search results weren't good and so that visibility quickly disappeared. Going further back, AlltheWeb was actually pretty decent but was eventually bought by Overture and then Yahoo and ended up in their graveyard. For everyone else it's…

I forgot about Cuil! I really wanted to like it.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#265

Post by shmeeed »

walt_grata wrote: post #249
shmeeed wrote: post #221

Earlier quoted context omitted.

I get the point, we've all been burnt. But if you're not trusting anybody anyway, why would explicity in a non-binding blog post / press release soothe you? We're in the middle of an AI bubble propping up the whole friggin US economy all by itself, driven mostly by a company that claimed to be a non-profit until a few years ago.

Because I've been burned by every big tech company I can think of. As for why it would be soothing, well because it gives me hope that when I read any further legal docs they'll hold to the post. What does ai have to do with this? The sooner that bubble bursts the better IMO.

The (Open)AI example was to illustrate the fact that companies can lie about their long-term plans or just change them whenever. They routinely do. And everybody who believed yesteryear's mission statement will then wind up feeling pretty stupid.

I'm considering Kagi a strategic ally in the fight against big tech right now.

It isn't a big tech company (yet). They don't have much of a moat either. Therefore, for the foreseeable future, they will be absolutely dependent on aligning their behaviour with their customer's interests, lest they lose them and go out of business.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#266

Post by SyneRyder »

carte_blanche wrote: post #250
SyneRyder wrote: post #128

Earlier quoted context omitted.

+1 so much for this. I have been doing the same, an SQLite database of my "own personal internet" of the sites I actually need. I use it as a tiny supplementary index for a metasearch engine I built for myself - which I actually did to replace Kagi. Building a metasearch engine is not hard to do (especially with AI now). It's so liberating when you control the ranking algorithm, and can supplement what the big engine…

Do you have any documentation/blog post for this? I would love to do something similar for my own use.

Unfortunately I still haven't had a chance to make a blog post about this, which I really must do. But I can give some quick hints. Anyone reading this, feel free to reach out & I can try to answer questions, and they might help my blog post too.

I started off with a meta-search calling out to Brave / Mojeek / Marginalia, and the basics of that are something that you can ask an AI to make for you as a one-file PHP script. I still think this is a good place to start, because you'll quickly find "okay, I can replace my everyday search engine with this". Once you're dogfooding your engine every day, you'll notice all the rough points you want to improve.

Once you've got an array of objects with Title, URL, Description, and splitting the URL into domain, TLD, subdomain, path, file extension... there's a lot of ranking you can apply just to those. Honestly, a lot of my "ranking" has just been applying increased rankings to domains that I visit most often. I have an array of about 600 domains that it applies ranking boosts to. You can try experimenting with your re-ranking there, before even starting to build your own index.

As for building your own (small, personal) index, the technical details are not as difficult as you'd think. An SQLite database file, that your PHP file reads, will take you a long way... especially if you enable FTS5 indexing. I only did that last week, and I should have done that at the beginning. Search times are 10ms, and not just on my personally curated index of 80,000 pages... I just added a 2nd database with 1.3 Million entries from DMOZ (the old Mozilla Directory), and it's still only about 10ms. My search engine now feels super fast when it gets results from my database. And when it finds zero results, it automatically falls back to the metasearch.

At 1.3 Million entries, the two databases are only about 550MB total. It's running on a shared hosting account and apparently they're not worried - but it's only available to me, so I'm only hitting it maybe 50 times a day maximum. I'll move it onto a VPS eventually, but every time I think "this must be using up too many resources", I find I'm thinking too small by at least a factor of 10x.

For getting started with PHP & SQLite, I found this blog post helpful - but at this point, your AI can vibe code the entire thing for you:

https://davidejones.com/blog/simple-site-search-engine-php-s...

It's amazing how far you can get with just SQLite and FTS5 and a little PHP. Read the Marginalia blog too, there's so much good information in there.

Don't hold yourself back, don't think it's impossible.

Post reply on HN