381–390 of 492 posts
Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…
lmao, hopefully the C code isn't nearly as bad as your html
Earlier quoted context omitted.
Especially since Cloudflare went public back in 2019, at which point any investors cashed out. - Sincerely, a Google employee who has nothing to do with the investment branch of the company
> at which point any investors cashed out. Well, actually that is also not true. At IPO preferred stocks convert to common but the investors can keep their ownership, they can but don't have to cash out or can only partially cash out. Investors can also keep board seats in many (or most?) cases.
Earlier quoted context omitted.
I really like GigaBlast. I wrote a "meta" search utility for myself that can query multiple search engines from the command line.^1 It mixes the results into a simplified SERP ("metaSERP"), optimised for a text-only browser, with indicators to show which search engine each result came from. The key feature is that it allows for what I might call "continuation searches". Each metaSERPs contains timestamps in its page…
Wow, do you happen to have published your utility so that other people can play with it?
If I see people on HN complain about how few results they get from search engines, then that could provide some motivation to publish. I am just not sure this is a problem for others besides me.
Many results I get from search engines are garbage. By creating a metaSERP with a much higher number of results overall, from a variety of sources, I believe I get a higher number of quality ones.
Earlier quoted context omitted.
Trusted curators is a dangerous dependency
How about just a meritocratic rating? Even here on HN I would appreciate some sort of weight on expert/experienced opinion. Although in theory I like the idea that every thought is judged on its own, the context of the author is more relevant the deeper the subject. That's one of the reasons I still read https://lobste.rs . It has a niche audience with industry experience.
Earlier quoted context omitted.
How about just a meritocratic rating? Even here on HN I would appreciate some sort of weight on expert/experienced opinion. Although in theory I like the idea that every thought is judged on its own, the context of the author is more relevant the deeper the subject. That's one of the reasons I still read https://lobste.rs . It has a niche audience with industry experience.
> meritocratic rating That is literally PageRank.
Earlier quoted context omitted.
You are going to pay for a generalized web search when DDG/Google/Bing/etc are free?
Yes. I use Brave Search and I hope they add a paid tier, which I think they have confirmed they'll add at a later date. If you don't pay, you are the product. Simple as that.
Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…
Earlier quoted context omitted.
Any open voting system is going to be under serious SEO pressure. That’s the real issue, Google has indirectly infected the web with junk sites optimized for it. Any new search engine now has a huge hurdle to sort through all the junk and if it succeeds the SEO industry is just going to target them. A more robust approach is simply pay people to evaluate websites. Assuming it costs say 2$ per domain to either whiteli…
Adding a gatekeeper collecting rent isn't a solution - the people using SEO are already spending money to get their name up high on the list.
More abstractly SEO is inherently a problem for search engines. Algorithms have no inherent way to separate clusters of websites setup to fake relevance from actually relevant websites. Personally I would exclude Quora from all search results, but even getting to the point your trying to make that kind of assessment is extremely difficult in the modern web. Essentially the minimum threshold for usefulness has become quite high which is a problem as Google continues to degenerate into uselessness.
Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…