Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Traceloop (YC W23) – Detecting LLM Hallucinations with OpenTelemetry

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 76 posts

Re: Launch HN: Traceloop (YC W23) – Detecting LLM Hallucinations with OpenTelemetry

#61

Post by roh26it »

lmeyerov wrote: post #55

As users of otel, we are looking at reusing otel for our LLM stack, and as it is easy to instrument, don't need a new framework for that part. However, the more interesting part is the storage: Imagine ingesting 100pg PDFs or 1M tweets, and doing many/big LLM map/reduce with big (128K+) context. In observability land, we generally have small payloads, sample data, and retire data... and backends + pricing assumes tha…

At Portkey, this is a problem we deal with quite a bit. Also the reason that Datadog and the traditional observability vendors did not work for LLM use cases since they're not built to handle large volumes of data.

We've done this through a careful combination of Clickhouse + MinIO for fast retrieval of log items + selected retrieval from the MinIO buckets.

Cost becomes a very big factor when managing, filtering and searching through TBs of data even for fairly small use cases.

One thing we lost in the process is full-text search over the request & response pairs and while we try to intelligently add metadata to requests to make searching easier, it isn't the complete experience yet. Still WIP as a problem statement to solve and maybe the last straw here. Any suggestions?

Re: Launch HN: Traceloop (YC W23) – Detecting LLM Hallucinations with OpenTelemetry

#62

Post by nirga »

lmeyerov wrote: post #55

As users of otel, we are looking at reusing otel for our LLM stack, and as it is easy to instrument, don't need a new framework for that part. However, the more interesting part is the storage: Imagine ingesting 100pg PDFs or 1M tweets, and doing many/big LLM map/reduce with big (128K+) context. In observability land, we generally have small payloads, sample data, and retire data... and backends + pricing assumes tha…

You're right. We faced those same issues. So we plan to move those prompts and completions to be sent as log events with some reference to the trace/span and not actually on the span.

The span can then only contain the most important data like the prompt template, model that was used, token usage, etc. You can then split the metadata (spans and traces) and the large payloads (prompts + completions) to different data stores.

Re: Launch HN: Traceloop (YC W23) – Detecting LLM Hallucinations with OpenTelemetry

#63

Post by ec109685 »

sakjur wrote: post #59
altdataseller wrote: post #57

I dont know if they are just casually using logos on their homepage but why the heck would Google (and even IBM) be using a product like this? Like… your entire future depends on getting this right and youre using a startup with 2-5 people to do this for you?!! Make it make sense..

If that’s true, they should want to evaluate all the options out there to ensure they’re not missing out. Though I think it’s more likely there’s some Googler who happen to use this service, note how the wording is “Engineers […] use our products[…]” rather than “Companies”.

Someone with a gmail address most likely.

Re: Launch HN: Traceloop (YC W23) – Detecting LLM Hallucinations with OpenTelemetry

#64

Post by nirga »

navaed01 wrote: post #51

Thank you for spending your time on something that is a barrier to AI adoption. Can you talk about your detection rates? False positives and false negatives. Perhaps you are still figuring this out I’m not sure why so many folks are being so derisive on this post.

Thanks! It can vary greatly between use cases - but we've seen extremely high detection rates for tagged texts (>95%). When switching to production, this gets trickier since you don't know what you don't know (so it's hard to tell how many "bad examples" we're missing). Our false positive rate (number of examples that were tagged as bad but weren't) has been around 2-3% out of the overall examples tagged as bad (positive) and we always work on decreasing this.

Re: Launch HN: Traceloop (YC W23) – Detecting LLM Hallucinations with OpenTelemetry

#65

Post by autonomousErwin »

nonameiguess wrote: post #50

This is poorly worded. Detecting "hallucinations" as the term is commonly used, as in a model making up answers not actually in its source text, or answers that are generally untrue, is fundamentally impossible. Verifying the truth of a statement requires empirical investigation. It isn't a feature of language itself. This is just the basic analytic/synthetic distinction identified by Kant centuries ago. It's why we…

Could you not detect likely hallucinations by running the same prompt multiple times between different models and looking at the vector divergence between the outputs? Kind of like an agreement between say GPT, Llama, other models which all agree - yes, this is likely a hallucination.

It's not 100% but enough to basically say to the human: "hey, look at this".

Re: Launch HN: Traceloop (YC W23) – Detecting LLM Hallucinations with OpenTelemetry

#66

Post by nirga »

autonomousErwin wrote: post #65
nonameiguess wrote: post #50

This is poorly worded. Detecting "hallucinations" as the term is commonly used, as in a model making up answers not actually in its source text, or answers that are generally untrue, is fundamentally impossible. Verifying the truth of a statement requires empirical investigation. It isn't a feature of language itself. This is just the basic analytic/synthetic distinction identified by Kant centuries ago. It's why we…

Could you not detect likely hallucinations by running the same prompt multiple times between different models and looking at the vector divergence between the outputs? Kind of like an agreement between say GPT, Llama, other models which all agree - yes, this is likely a hallucination. It's not 100% but enough to basically say to the human: "hey, look at this".

You can do it and it's a good way of doing that - from our experiments that can catch most errors. You don't even need to use different models - even using the same model (I don't mean asking "are you sure?" - just re-running the same workflow) will give you nice results. The only problem is that it's super expensive to run it on all your traces so I wouldn't recommend that as a monitoring tool.

Re: Launch HN: Traceloop (YC W23) – Detecting LLM Hallucinations with OpenTelemetry

#67

Post by suyash »

R21M1214 wrote: post #24
swyx wrote: post #8

Earlier quoted context omitted.

not to mention langsmith? braintrust? humanloop? does that count? not sure what else - lets crowdsource a list here so that people can find them in future

Im not sure which ones are Otel compliant. Im only aware of 3 that are Otel compliant: 1. Traceloop Otel 2. Langtrace.ai Otel 3. OpenLIT Otel 4. Portkey 5. Langfuse 6. Arize LLM 7. Phoniex SDK 8. Truera LLM 9. Truelens 10. Context 11. Braintrust 12. Parea 13. Context AI 14. openlayer.com 15. Deepchecks 16. langsmith 17. Confident AI 18. Helicone 19. Langwatch.ai 20. Arthur 21. Aporia 22. scale.com 23. Whylabs 24. gen…

This is a great list, I'm planning on writing some sample apps and blogs about OpenTelemetry for LLM's and this will be helpful. Which are the most popular open source ones amongst these?

Re: Launch HN: Traceloop (YC W23) – Detecting LLM Hallucinations with OpenTelemetry

#68

Post by lmeyerov »

roh26it wrote: post #61
lmeyerov wrote: post #55

As users of otel, we are looking at reusing otel for our LLM stack, and as it is easy to instrument, don't need a new framework for that part. However, the more interesting part is the storage: Imagine ingesting 100pg PDFs or 1M tweets, and doing many/big LLM map/reduce with big (128K+) context. In observability land, we generally have small payloads, sample data, and retire data... and backends + pricing assumes tha…

At Portkey, this is a problem we deal with quite a bit. Also the reason that Datadog and the traditional observability vendors did not work for LLM use cases since they're not built to handle large volumes of data. We've done this through a careful combination of Clickhouse + MinIO for fast retrieval of log items + selected retrieval from the MinIO buckets. Cost becomes a very big factor when managing, filtering and…

Clickhouse has text + vector indexes, so that may be native, though we have never used them and I find vector indexes tricky to scale w other DBs. Text... Or neither... may be enough in practice tho as we mostly only care about searching on metadata dimensions like task.

We are thinking about sampled hot data for ops staff in otel DB+UIs, and long-term full data in S3/Clickhouse for custom tooling. It'd be cool if we could send Clickhouse historical otel sessions to grafana etc on demand, but likely a bridge too far...

Re: Launch HN: Traceloop (YC W23) – Detecting LLM Hallucinations with OpenTelemetry

#69

Post by nirga »

lmeyerov wrote: post #68
roh26it wrote: post #61

Earlier quoted context omitted.

At Portkey, this is a problem we deal with quite a bit. Also the reason that Datadog and the traditional observability vendors did not work for LLM use cases since they're not built to handle large volumes of data. We've done this through a careful combination of Clickhouse + MinIO for fast retrieval of log items + selected retrieval from the MinIO buckets. Cost becomes a very big factor when managing, filtering and…

Clickhouse has text + vector indexes, so that may be native, though we have never used them and I find vector indexes tricky to scale w other DBs. Text... Or neither... may be enough in practice tho as we mostly only care about searching on metadata dimensions like task. We are thinking about sampled hot data for ops staff in otel DB+UIs, and long-term full data in S3/Clickhouse for custom tooling. It'd be cool if we…

I think you can (pretty) easily set this up with an otel collector and something that replays data from S3 - there's a native implementation that converts otel to clickhouse
Post reply on HN