Viewing profile — nirga

Hey! We don’t support an otel collector directly - you have to connect it to some backend. Minimal one can be jaeger for example

Sorry didn’t know that. It’s not my article so didn’t want to attribute the title to myself

For example - the fact that the FE environment variables are hardcoded at build time makes it hard to just deploy a container

I think that's the key benefit of using OpenTelemetry - it's pretty efficient and the performance footprint is negligible.

Thanks for spotting those! We'll fix it asap

I think you can (pretty) easily set this up with an otel collector and something that replays data from S3 - there's a native implementation that converts otel to clickhouse

You can do it and it's a good way of doing that - from our experiments that can catch most errors. You don't even need to use different models - even using the same model (I don't …

Thanks! It can vary greatly between use cases - but we've seen extremely high detection rates for tagged texts (>95%). When switching to production, this gets trickier since you do…

You're right. We faced those same issues. So we plan to move those prompts and completions to be sent as log events with some reference to the trace/span and not actually on the sp…

Thanks! I wasn’t offended or anything, don’t get the wrong impression. What strikes me odd is the fact that an AI that checks AI is an issue. Because AI can mean a lot of things - …

It has the same logic of saying you dont want to use a computer to monitor or test your code since it will mean that a computer will monitor a computer. AI is a broad term, I agree…

I replied to you in a different thread, I don't think calling our companies "deceptive" will help you or me get anywhere. While I agree with you that detection will never be hermet…

I'm sorry but this is not what we do. We don't use LLMs to grade your LLM calls.

I think that LLMs are hallucinating by design. I'm not sure we'll ever get to a 0% hallucinations and we should be ok with it (at least for the next coming years?). So getting an a…

I think it depends on the use case and how you define hallucinations. We've seen our metrics perform well (=correlates with human feedback) for use cases like summarization, RAG qu…

Ping me over slack (traceloop.com/slack) or email nir at traceloop dot com

roger that! I like them though (am I a normie then?)

I tend to find classic NLP metric more predictable and stable than "LLM as a judge" metrics so I'd try to see if you rely on them more. We've written a couple of blog posts about s…

We trained our own models for some of them, and we combined some well known NLP metrics (like Gruen [1]) to make this work. You're right that it's hard to figure out how to "trust"…