101–110 of 118 posts
I can’t believe programming is now at the stage where advice like "first have the computer give you totally wrong answers, then just find a function that maps the wrong answers to the correct ones!" is a thing.
Earlier quoted context omitted.
I feel like I'm fighting a losing battle but I don't see why so many people consider LLMs innately non-deterministic, an LLM running on a CPU with greedy decoding would be 100% deterministic and still have the intelligence we associate with LLMs
> I feel like I'm fighting a losing battle but I don't see why so many people consider LLMs innately non-deterministic ... Because LLMs are artificial neural networks[0] (ANN), which are statistical in nature, and thus intrinsically non-deterministic. Pretty much every AI algorithm has randomness involved in its definition and many (most?) incorporate probabilities. 0 - https://en.wikipedia.org/wiki/Neural_network_(m…
Earlier quoted context omitted.
That’s one of the properties that distinguishes intelligence from traditional deterministic algorithms.
I feel like I'm fighting a losing battle but I don't see why so many people consider LLMs innately non-deterministic, an LLM running on a CPU with greedy decoding would be 100% deterministic and still have the intelligence we associate with LLMs
Earlier quoted context omitted.
> I feel like I'm fighting a losing battle but I don't see why so many people consider LLMs innately non-deterministic ... Because LLMs are artificial neural networks[0] (ANN), which are statistical in nature, and thus intrinsically non-deterministic. Pretty much every AI algorithm has randomness involved in its definition and many (most?) incorporate probabilities. 0 - https://en.wikipedia.org/wiki/Neural_network_(m…
Now that you mention it, I think "statistical" might be a good word choice as it probably closely matches what people mean when they say an LLM is nondeterministic. However, on a technical level, neural network inference truly is inherently deterministic. The only nondeterminism in LLMs is parallelism-dependency in the way floating point operations are carried out and in the decoding strategy
Fixing the seed is still intentional bias. Or you could force it to always take the one token with the highest probability, but that is still biased sampling. Deterministic, sure, but intentionally wrong just to avoid a technically
Earlier quoted context omitted.
Now that you mention it, I think "statistical" might be a good word choice as it probably closely matches what people mean when they say an LLM is nondeterministic. However, on a technical level, neural network inference truly is inherently deterministic. The only nondeterminism in LLMs is parallelism-dependency in the way floating point operations are carried out and in the decoding strategy
only if you bias your "random sampling of the distributions it gives" Fixing the seed is still intentional bias. Or you could force it to always take the one token with the highest probability, but that is still biased sampling. Deterministic, sure, but intentionally wrong just to avoid a technically
Earlier quoted context omitted.
Unironically, LLM is absolutely amazing at giving you the right answers when you put in wrong input, compared to every other algorithm ever invented.
Yeah they solved the main issue I had with Google. "I remember this exact thing, but not exactly how it was phrased..." Except not as much as I'd like... they often also don't know what the hell I'm talking about, and it still takes them twenty minutes of Googling to find the right page!
It nearly always works.
New embedding models support queries, so you don’t need to hallucinate a document before finding the nearest neighbor. Curious how it compares to this approach since you’d get to skip the LLM altogether.
What does it mean that an "embedding model supports queries"? An embedding model maps text to embedding vectors. You can always perform queries with such embedding vectors against a stored set of embeddings.
Which accomplishes the same thing as HyDE, but in the model instead of in text space. If the encoder has a query mode you skip the rewrite.
Voyage has an example of this in `input_type` https://docs.voyageai.com/reference/embeddings-api
I don't know the exact syntax any more, but I expect this could be solved by a single sql query that uses "inexact but close" queries and a bunch of indexes (and perhaps tags) on each category.
Like some sort of Jaccard Index based on how many tags are shared? https://en.wikipedia.org/wiki/Jaccard_index
[1] https://www.postgresql.org/docs/current/textsearch-intro.htm...
[2] https://www.elastic.co/docs/explore-analyze/query-filter