Viewing profile — _jonas

Evals are critical, and I love the practicality of this guide! One problem not covered here is: knowing which data to review. If your AI system produces say 95% accurate responses,…

You might be thinking of LLM as-a-judge, where one simply asks another LLM to fact-check the response. Indeed that is very unreliable due to LLM hallucinations, the problem we are …

This is why I built a startup for automated real-time trustworthiness scoring of LLM responses: https://help.cleanlab.ai/tlm/ Tools to mitigate unchecked hallucination are critical…

Exactly, that's why my startup recommends all LLM outputs should come with trustworthiness scores: https://cleanlab.ai/tlm/

My startup is working on this fundamental problem. You can try out our early product here: https://cleanlab.ai/tlm/ (free to try, we'd love to hear your feedback)

I see this fallacy often too. My company provides hallucination detection software: https://cleanlab.ai/tlm/ But we somehow end up in sales meetings where the person who requested …

[dead]

Has anyone run any meaningful benchmarks of this vs. google vs. perplexity?

This one looks pretty good, haven't tried it yet though: https://github.com/QuivrHQ/quivr

It's fun to try and guess what semantic concepts might be captured within individual dimensions / pairs of dimensions of the embeddings space.

Curious to learn how much harder it is to red-team models that use the second line of defense of an explicit guardrails library that checks the LLM response in a second step. Such …

I'm excited for LLM applications that can setup, monitor/validate, and optimize data pipelines at scale. Seems possible soon given that SQL and most data records aren't intended to…

It's easier to find the data now, I've run some benchmarks on it. Great to see OpenAI open-sourcing datasets like this!

[dead]

Here are some benchmarks I ran that compare the precision/recall of various LLM error-detection methods, including logprobs and LLM self-evaluation / verbalized confidence: https:/…

There is however a subfield of statistical ML of model uncertainty quantification. I've developed a product by applying to it to LLMs that can score the trustworthiness of any LLM …

[dead]

To try out an existing product that quantifies LLM uncertainty (accurately incorporating both aleatoric & epistemic uncertainty), you can try this Trustworthy Language Model I buil…

New feature alert: Auto-train & deploy reliable ML models (more accurate than fine-tuned OpenAI LLMs) on messy real-world data — all in just a few clicks! Common reasons companies …

[dead]

AutoGluon is an easy-to-use AutoML toolkit for deep learning that allows you to automatically leverage state-of-the-art techniques. Writing barely any code, we recently used AutoGl…