Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: You don't need to adopt new tools for LLM observability

github.com

Post reply on HN
Print view View original

1–10 of 24 posts

Show HN: You don't need to adopt new tools for LLM observability

#1

Post by tomerf2 »

If you've built any web-based app in the last 15 years, you probably used something like Datadog, New Relic, Sentry, etc. to monitor and trace your app, right?

Why should it be different when the app you're building happens to be using LLMs?

So today we're open-sourcing OpenLLMetry-JS. It's an open protocol and SDK, based on OpenTelemetry, that provides traces and metrics for LLM JS/TS applications and can be connected to any of the 15+ tools that already support OpenTelemetry. Here's the repo: https://github.com/traceloop/openllmetry-js

A few months ago we launched the python flavor here (https://news.ycombinator.com/item?id=37843907) and we've now built a compatible one for Node.js.

Would love to hear your thoughts and opinions!

Check it out -

Docs: https://www.traceloop.com/docs/openllmetry/getting-started-t...

Github: https://github.com/traceloop/openllmetry-js https://github.com/traceloop/openllmetry

Show HN: You don't need to adopt new tools for LLM observability
github.com

Re: Show HN: You don't need to adopt new tools for LLM observability

#2

Post by tomgs »

Cool! Two questions:

1. Where do you see this observability for LLM thing going? What's the end game? Is it like in traditional observability where all formats eventually will converge to one format (which OpenTelemetry is trying to be)? I feel it might be a little bit early to tell, tho

2. I noticed you do auto-detection of the framework used, like LLamaIndex et al. Except for annotations, is there a deeper connection to the LLM framework used? This is auto-instrumentation, so I assume you do most of the heavy lifting, but should users of this framework expect some cool hidden eggs when they look at their telemetry?

Re: Show HN: You don't need to adopt new tools for LLM observability

#3

Post by tomerf2 »

tomgs wrote: post #2

Cool! Two questions: 1. Where do you see this observability for LLM thing going? What's the end game? Is it like in traditional observability where all formats eventually will converge to one format (which OpenTelemetry is trying to be)? I feel it might be a little bit early to tell, tho 2. I noticed you do auto-detection of the framework used, like LLamaIndex et al. Except for annotations, is there a deeper connecti…

Thanks!

1. Huh, good question. Hopefully there will be convergence. We started discussing with other companies in this domain to support or even switch to OpenTelemetry.

2. Nothing specific, except for - as you mentioned - being able to see trace of a RAG pipeline automatically.

Re: Show HN: You don't need to adopt new tools for LLM observability

#4

Post by tomgs »

tomerf2 wrote: post #3
tomgs wrote: post #2

Cool! Two questions: 1. Where do you see this observability for LLM thing going? What's the end game? Is it like in traditional observability where all formats eventually will converge to one format (which OpenTelemetry is trying to be)? I feel it might be a little bit early to tell, tho 2. I noticed you do auto-detection of the framework used, like LLamaIndex et al. Except for annotations, is there a deeper connecti…

Thanks! 1. Huh, good question. Hopefully there will be convergence. We started discussing with other companies in this domain to support or even switch to OpenTelemetry. 2. Nothing specific, except for - as you mentioned - being able to see trace of a RAG pipeline automatically.

A quick one :)

While we're on the topic - how does traceloop factor into all of this? What's the connection between the two? I assume the former is the LLM observability platform (Datadog for LLM?) and the latter is your own auto-instrumentation thingie to supplement it?

Re: Show HN: You don't need to adopt new tools for LLM observability

#5

Post by nirga »

tomgs wrote: post #4
tomerf2 wrote: post #3

Earlier quoted context omitted.

Thanks! 1. Huh, good question. Hopefully there will be convergence. We started discussing with other companies in this domain to support or even switch to OpenTelemetry. 2. Nothing specific, except for - as you mentioned - being able to see trace of a RAG pipeline automatically.

A quick one :) While we're on the topic - how does traceloop factor into all of this? What's the connection between the two? I assume the former is the LLM observability platform (Datadog for LLM?) and the latter is your own auto-instrumentation thingie to supplement it?

Yes, Traceloop is kind of a Sentry for LLMs

Re: Show HN: You don't need to adopt new tools for LLM observability

#7

Post by nirga »

lmeyerov wrote: post #6

Re:python, if we are already doing otel, how would this interop? Eg, if we don't want to break our current imports, and control where the new instrumentation goes (Fwiw, This is a great direction!)

Super easy - you can just use the standalone instrumentations directly - https://www.traceloop.com/docs/openllmetry/tracing/without-s...

Re: Show HN: You don't need to adopt new tools for LLM observability

#9

Post by epistasis »

I was looking to see what the actual metrics would be for a completion, to see if this is something of interest to me. So I tried to run the example here:

https://www.traceloop.com/openllmetry

Problem 1 (very minor): it's missing an `import os`

Problem 2: I need an API key.

Problem 3: The link that it tells me to go to for an API key is malformed: https://https//app.traceloop.com/settings/api-keys

Is there a way to see what the output is like without getting an account, and presumably also connecting to an observability platform like Grafana? I already made a venv and installed the package, so I'm not sure if I'm ready for even more steps just to see if this is something that might be useful to me.

Re: Show HN: You don't need to adopt new tools for LLM observability

#10

Post by nirga »

epistasis wrote: post #9

I was looking to see what the actual metrics would be for a completion, to see if this is something of interest to me. So I tried to run the example here: https://www.traceloop.com/openllmetry Problem 1 (very minor): it's missing an `import os` Problem 2: I need an API key. Problem 3: The link that it tells me to go to for an API key is malformed: https://https//app.traceloop.com/settings/api-keys Is there a way to s…

Thanks for the issues - I'll fix it! :sweat_smile:

Reg. Grafana and others - it's simple, just set the env vars - https://www.traceloop.com/docs/openllmetry/integrations/intr...

Post reply on HN