61–70 of 124 posts
It's incredible. I can't believe it but it actually works quite nicely. If 10K $5 subscriptions can cover its cost, maybe a community run search engine funded through donations isn't that insane?
Models make it cheap to replicate and perform what tech companies do. Their insurmountable moats are lowering as we speak.
"There was one surprise when I revisited costs: OpenAI charges an unusually low $0.0001 / 1M tokens for batch inference on their latest embedding model. Even conservatively assuming I had 1 billion crawled pages, each with 1K tokens (abnormally long), it would only cost $100 to generate embeddings for all of them. By comparison, running my own inference, even with cheap Runpod spot GPUs, would cost on the order of 10…
I don’t think OpenAI train on data processed via the API, unless there’s an exception specifically for this.
Just wow. My greatest respect! Also an incredible write up. I like the take-away that an essential ingredient to a search engine is curated and well filtered data (garbage in garbage out) I feel like this has been a big learning of the LLM training too, rather work with less much higher quality data. I'm curious how a search engine would perform where all content has been judged by an LLM.
I'm currently trying to get a friends small business website to rank. I have a decent understanding of SEO, doing more technically correct things and did a decent amount of hand written content specific to local areas and services provided. Two months in, bing still hasn't crawled the fav icon. Google finally did after a month. I'm still getting outranked by tangentially related services, garbage national lead collec…
It's incredible. I can't believe it but it actually works quite nicely. If 10K $5 subscriptions can cover its cost, maybe a community run search engine funded through donations isn't that insane?
I think even more spectacularly, we may be witnessing the feature to feature obsolescence of big tech. Models make it cheap to replicate and perform what tech companies do. Their insurmountable moats are lowering as we speak.
Just wow. My greatest respect! Also an incredible write up. I like the take-away that an essential ingredient to a search engine is curated and well filtered data (garbage in garbage out) I feel like this has been a big learning of the LLM training too, rather work with less much higher quality data. I'm curious how a search engine would perform where all content has been judged by an LLM.
I'm currently trying to get a friends small business website to rank. I have a decent understanding of SEO, doing more technically correct things and did a decent amount of hand written content specific to local areas and services provided. Two months in, bing still hasn't crawled the fav icon. Google finally did after a month. I'm still getting outranked by tangentially related services, garbage national lead collec…
Do you have any backlinks? If not, it’s working as intended?
It's incredible. I can't believe it but it actually works quite nicely. If 10K $5 subscriptions can cover its cost, maybe a community run search engine funded through donations isn't that insane?
It's been clear to anyone familiar with encoder only LLMs that Google is effectively dead. The only reason why it still lives is that it takes a while to crawl the whole web and keep the index up to date. If someone like common crawl, or even a paid service, solves the crawling of the web in real time then the moat Google had for the last 25 years is dead and search is commoditized.
Earlier quoted context omitted.
It's been clear to anyone familiar with encoder only LLMs that Google is effectively dead. The only reason why it still lives is that it takes a while to crawl the whole web and keep the index up to date. If someone like common crawl, or even a paid service, solves the crawling of the web in real time then the moat Google had for the last 25 years is dead and search is commoditized.
The team that runs the Common Crawl Foundation is well aware of how to crawl and index the web in real time. It's expensive, and it's not our mission. There are multiple companies that are using our crawl data and our web graph metadata to build up-to-date indexes of the web.
But you are pretty much the only people who can save the web from AI bots right now.
The sites I administer are drowning in bots, and the applications I build which need web data are constantly blocked. We're in the worst of all possible worlds and the simplest way to solve it is to have a middleman that scrapes gently and has the bandwidth to provide an AI first API.
Earlier quoted context omitted.
The team that runs the Common Crawl Foundation is well aware of how to crawl and index the web in real time. It's expensive, and it's not our mission. There are multiple companies that are using our crawl data and our web graph metadata to build up-to-date indexes of the web.
Yes, I've used your data myself on a number of occasions. But you are pretty much the only people who can save the web from AI bots right now. The sites I administer are drowning in bots, and the applications I build which need web data are constantly blocked. We're in the worst of all possible worlds and the simplest way to solve it is to have a middleman that scrapes gently and has the bandwidth to provide an AI fi…
"There was one surprise when I revisited costs: OpenAI charges an unusually low $0.0001 / 1M tokens for batch inference on their latest embedding model. Even conservatively assuming I had 1 billion crawled pages, each with 1K tokens (abnormally long), it would only cost $100 to generate embeddings for all of them. By comparison, running my own inference, even with cheap Runpod spot GPUs, would cost on the order of 10…
I don’t think OpenAI train on data processed via the API, unless there’s an exception specifically for this.
Mad respect. This is an incredible project to pull together all these technologies. The crown jewel of a search engine is its ranking algorithm. I'm not sure how LLM is being used in this regard in here. One effective old technique for ranking is to capture the search-to-click relationship by real users. It's basically the training data by human mapping the search terms they entered to the links they clicked. With ju…
That's not very effective. Ever heard of clickbait?
Like I've said uncountable times before, the only effective technique to clean out the search results of garbage is to use a point system that penalises each 3rd party advertisement placed on the page.
The more adverts, the lower the rank.
And the reason that will work is because you are directly addressing the incentive for producing garbage - money!
The result should be "when two sites have the same basic content, in the search results promote the one without ads over the ones with ads".
Until this is done, search engines will continue serving garbage, because they are rewarding those actors who are producing garbage.