Viewing profile — gbmatt

Only Big Tech (Microsoft,Google,Facebook) can crawl the web at scale because they own the major content companies and they severly throttle the competition's crawlers, and sometime…

We are just robots in a human simulator, reliving our creation.

Q: how might an AI algorithm be modified in order to return citations with its response? A: There are several ways in which an AI algorithm could be modified to return citations wi…

I just posted this same comment on the ddg story, but I'm going to post it here as well. Google forced my search engine (gigablast) basically out of business. I had ixquick.com as …

Yeah, Google forced my search engine basically out of business. I had ixquick.com as a big client at one time; I was providing them with search results from my custom web search en…

everyone needs equal access to public data. right now only big tech can download the many web pages (without thottling or being ip banned) on linkedin (microsoft), youtube (google)…

the complexity of the search algorithm has also increased substantially since 2005 And, in 2005, a billion page index was pretty big. Now it's closer to 100 billion.

thanks ben, you are too kind.

the javascript is run by your browser, so you can fully audit it.

hey thanks for the recognition, people. :) finally, all my problems are solved. this comment is here for hacker news karma points.

I'd argue that a level playing field and more competition in the search space is a good thing.

100% custom.

that's tripped out. where did you hear about that?

I'll admit I had not been working on the quality of single term queries as much as I should have lately. However, especially for such simple queries, having a database of link text…

Cloudflare is not the only gatekeeper, too. Keep that in mind. There's many others and, as an upstart search engine operator, it's quite overwhelming to have to deal with them all.…

It's not quite that easy. Have you ever tried it? See my post below. Basically, yes, I've done it, but i had to go through a lot and was lucky enough to even get them to listen to …

So Brave is still dependent on Google and Bing it seems. Also is this Brave's CEO: https://www.bbc.com/news/technology-26868536 https://www.nytimes.com/2020/12/22/business/brave-br…

there's some stuff here : https://github.com/gigablast/open-source-search-engine

brave 'falls back' to bing. which in my experience is most of the time. in fact, out of all the queries i did a while back, they all seemed to come directly from bing. is there a w…

yes, large proxy networks are potential solutions. but they cost money, and you are playing a cat and mouse game with turing tests, and some sites require a login. furthermore, peo…

it's both storage and computational. they go hand in hand.

both ddg and brave are bing (microsoft) in disguise.

it's continually spidering. just not at a high rate. actually, back in the day i had real time updates while google was doing the 'google dance'. that caused quite a stir in the we…