Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Building a web search engine from scratch with 3B neural embeddings

blog.wilsonl.in

Post reply on HN
Print view View original

81–90 of 124 posts

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#81

Post by creesch »

ricardobeat wrote: post #46
poly2it wrote: post #40

Very cool project! Just out of interest, I sent a query I've had difficulties getting good results for with major engines: "what are some good options for high-resolution ultrawide monitors?". The response in this engine for this query at this point seems to have the same fallacy as I've seen in other engines. Meta-pages "specialising" in broad rankings are preferred above specialist data about the specific sought-af…

You could argue that it is not really a search query. There is not a particular page that answers the question “correctly”, it requires collating multiple sources and reasoning . That is not a search problem.

This argument almost feels disingenuous to me. Of course, there isn't going to be one resource that will completely answer the question. However, there are going to be resources that are much likelier to contain correct parts to the answer and there are resources that are much likely to contain just SEO fluff.

The whole premise of what makes a good search engine has been based on the idea of surfacing those results that most likely contain good information. If that was not the case Google would not have risen to such dominance in the first place.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#82

Post by gkbrk »

reasonableklout wrote: post #79
cedws wrote: post #71

Earlier quoted context omitted.

Yes, it would be disastrous for OpenAI if it got out they are training on B2B data despite saying they don’t.

Have they said they don't? (actually curious)

Yes, they have. [1]

> Your data is your data. As of March 1, 2023, data sent to the OpenAI API is not used to train or improve OpenAI models (unless you explicitly opt in to share data with us).

[1]: https://platform.openai.com/docs/guides/your-data

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#83

Post by trhway »

cedws wrote: post #57
arkmm wrote: post #51

"There was one surprise when I revisited costs: OpenAI charges an unusually low $0.0001 / 1M tokens for batch inference on their latest embedding model. Even conservatively assuming I had 1 billion crawled pages, each with 1K tokens (abnormally long), it would only cost $100 to generate embeddings for all of them. By comparison, running my own inference, even with cheap Runpod spot GPUs, would cost on the order of 10…

I don’t think OpenAI train on data processed via the API, unless there’s an exception specifically for this.

i'd not rule out some approach like instead of training directly on the data, may be they would train on a very high dimensional embedding of such a data (or some other similarly "anonymized", yet still very semantically rich representation of the data)

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#84

Post by wilsonzlin »

demarq wrote: post #78
demarq wrote: post #73

The title should be “10x engineer creates Google in their spare time” But seriously what an amazing write up, plus animations, analysis etc etc. Bravo. It was also ironic to see AWS failing quite a few use cases here. Stuff to think about.

Also looking into the AWS limits; > SQS had very low concurrent rate limits that could not keep up with the throughput of thousands of workers across the pipeline. I could not find this perhaps the author meant Lambda limits? > services like S3 have quite low rate limits — there are hard limits, but also dynamic per-account/bucket quotas You have virtually unlimited throughput with prefix partitions

I'm not sure what were the exact limits, but I definitely recall running into server errors with S3 and the OCI equivalent service — not technically 429s but enough to essentially limit throughput. SQS had 429s, I believe due to number of requests and not messages, but they only support batching at most 10.

I definitely wanted these to "just work" out of the box (and maybe I could've worked more with AWS/OCI given more time), as I wanted to focus on the actual search.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#85

Post by zipy124 »

The author claims "It should be far less susceptible to keyword spam and SEO tactics." however anyone with a cursory knowledge of the limitations of embeddings/LLM's knows the hardest part is that there is no seperation between the prompt and the content to be queried (e.g "ignore all previous instructions" etc...). It would not be hard to adversarially generate embeddings for SEO, in-fact it's almost easier since you know the maths underlying the algorithm to fit to.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#86

Post by gunalx »

divineg wrote: post #44

It's incredible. I can't believe it but it actually works quite nicely. If 10K $5 subscriptions can cover its cost, maybe a community run search engine funded through donations isn't that insane?

Kagi seems to partially be that. Yes really corpo but way Better wibes than Google. Searxng is a bit diffrent but also a thing.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#87

Post by jll29 »

Not sure where you are based, but if you were in the EU and had no commercial intentions, you might want to consider adding the crawls from OpenWebSearch.eu, an EU-funded research project to provide an open crawl of a substantial part of the Web (they also collaborate with Common Crawl), its plain text and an index:

  https://openwebsearch.eu/
It would be fantastic if someone could provide a not-for-profit decent quality Web search engine.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#88

Post by binarymax »

zipy124 wrote: post #85

The author claims "It should be far less susceptible to keyword spam and SEO tactics." however anyone with a cursory knowledge of the limitations of embeddings/LLM's knows the hardest part is that there is no seperation between the prompt and the content to be queried (e.g "ignore all previous instructions" etc...). It would not be hard to adversarially generate embeddings for SEO, in-fact it's almost easier since yo…

Classic HN dismissive comment.

The talent displayed here is immense. I challenge you to do better.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#89

Post by demarq »

wilsonzlin wrote: post #84
demarq wrote: post #78

Earlier quoted context omitted.

Also looking into the AWS limits; > SQS had very low concurrent rate limits that could not keep up with the throughput of thousands of workers across the pipeline. I could not find this perhaps the author meant Lambda limits? > services like S3 have quite low rate limits — there are hard limits, but also dynamic per-account/bucket quotas You have virtually unlimited throughput with prefix partitions

I'm not sure what were the exact limits, but I definitely recall running into server errors with S3 and the OCI equivalent service — not technically 429s but enough to essentially limit throughput. SQS had 429s, I believe due to number of requests and not messages, but they only support batching at most 10. I definitely wanted these to "just work" out of the box (and maybe I could've worked more with AWS/OCI given mo…

Those are reasonable expectations. I’m very impressed with how it all worked out.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#90

Post by dweinus »

cedws wrote: post #71
dannyw wrote: post #69

Earlier quoted context omitted.

Can you truly trust them though?

Yes, it would be disastrous for OpenAI if it got out they are training on B2B data despite saying they don’t.

We're both talking about the company whose entire business model is built on top of large scale copyright infringement, right?
Post reply on HN