11–20 of 24 posts
What problem(s) does this solve? I have a ticket in my backlog. Your SDK unlocks the solution. What is that ticket's title? (I'm a bit thick, and need concrete examples for things to click.)
Earlier quoted context omitted.
A quick one :) While we're on the topic - how does traceloop factor into all of this? What's the connection between the two? I assume the former is the LLM observability platform (Datadog for LLM?) and the latter is your own auto-instrumentation thingie to supplement it?
Yes, Traceloop is kind of a Sentry for LLMs
I thought Observability in this context means the ability to introspectively make sense of why the LLM output what it did, which is a difficult problem because the model parameters are effectively an unintelligible morass of numbers. Does this help with that and if so how?
LLM Observability usually means: prompts and completions, which model was used, errors and exceptions (rate limits, network errors), as well as metrics (latency, output speed, time to first token when streaming, USD/token and cost breakdowns). All of this is well suited to be captured in the existing observability stack. OpenLLMetry makes this really easy and interoperable - chapeau.
In my view, observability is not the core value that solutions like Baserun, Athina, LangSmith, Parea, Arize, Langfuse (my project) and many others solve for. Developing a useful LLM application requires iterative workflows and tinkering. That's what these solutions help with and augment.
There are specific problems to building an LLM application such as managing/versioning of prompts, running evaluations, blending multiple different evaluation sources, collecting datasets to test/benchmark an application, helping with fine-tuning models on high-quality production completions, debugging root causes of quality/latency/cost issues, ...
Most solutions either replicate logs (LLM I/O) or traces at first, as they are a necessary starting point to then build solutions for the other workflow problems. As the observability piece gets more standardized over time, I can see how integrating with the standard makes a ton of sense. Always happy to chat about this.
What problem(s) does this solve? I have a ticket in my backlog. Your SDK unlocks the solution. What is that ticket's title? (I'm a bit thick, and need concrete examples for things to click.)
Fully agree - even as a founder of an ‘LLM observability company’. Observability does not need to be reinvented to get detailed traces/metrics/logs of the LLM part of an application. LLM Observability usually means: prompts and completions, which model was used, errors and exceptions (rate limits, network errors), as well as metrics (latency, output speed, time to first token when streaming, USD/token and cost breakd…
Would love to see you integrate and adopt this as soon as it makes sense to you. OpenTelemetry is a great and mature piece of technology and we should all be aligning around it now, while it’s still easy to do so.
I thought Observability in this context means the ability to introspectively make sense of why the LLM output what it did, which is a difficult problem because the model parameters are effectively an unintelligible morass of numbers. Does this help with that and if so how?
Pretty sure this just structures logs for requests to common 3rd party LLM providers. Which I guess is useful, but it's not some kind of problem unique to LLMs.
I was looking to see what the actual metrics would be for a completion, to see if this is something of interest to me. So I tried to run the example here: https://www.traceloop.com/openllmetry Problem 1 (very minor): it's missing an `import os` Problem 2: I need an API key. Problem 3: The link that it tells me to go to for an API key is malformed: https://https//app.traceloop.com/settings/api-keys Is there a way to s…
Thanks for the issues - I'll fix it! :sweat_smile: Reg. Grafana and others - it's simple, just set the env vars - https://www.traceloop.com/docs/openllmetry/integrations/intr...
I might revisit if a trusted friend tells me it's useful, but it will take that sort of recommendation in order for me to spend more time on this. Seems like some example outputs would the thing you should show, at least somewhere prominently.