Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: OpenLLMetry – OpenTelemetry-based observability for LLMs

github.com

Post reply on HN
Print view View original

11–20 of 35 posts

Re: Show HN: OpenLLMetry – OpenTelemetry-based observability for LLMs

#14

Post by nirga »

serverlessmom wrote: post #13

Pretty neat! I assume it's just measuring traces right now? Any plans to add some top level metrics like build times, prompt length, etc?

Yes, only traces for now. We do want to send out metrics for prompt length, token usage, etc. like you mentioned. Hopefully will be available soon (and we welcome contributions :) )

Re: Show HN: OpenLLMetry – OpenTelemetry-based observability for LLMs

#16

Post by nirga »

jeffchao wrote: post #15

Cool! It looks like you effectively do auto instrumentation. Have you found there to be interesting nuances between LLM providers? Tracing is great and trace aggrgegates (with context!) cross-vendor would be even more awesome.

Wow, where do I start? The APIs are not that similar, but we're trying to use the same set of semantic conventions for everyone so for example you'll always get the model version, or the temperature in the same attribute. Which kinda means it's identical cross-vendor, at least on the o11y side.

Here are all the semantic conventions we've defined so far - https://github.com/traceloop/openllmetry/tree/main/packages/...

Re: Show HN: OpenLLMetry – OpenTelemetry-based observability for LLMs

#17

Post by peter_d_sherman »

Great idea!

Observability (AKA, debug/proxy/statistics/logging/visualization layer) -- for LLM's (AKA Chat AI's)...

Hmmm, you know, I would love something for ChatGPT (and other AI chatbots) -- where you could open a second tab or window -- and see (and potentially interact with) debug info and statistics from prompts given to that AI in its main input window, in realtime...

Sort of like what Unix's STDERR is for programs running on Unix -- but an "AI STDERR" AKA debug channel, for AI's...

I'm guessing (but not knowing) that in the future, there will be standards defined for debug interfaces to AI's, standards defined for the data formats and protocols traversing those interfaces, and standards defined for such things as error, warning, hint, and informational messages...

Oh sure, a given AI company could pick a series of their own interfaces, data protocols and how to interpret that data.

But if so, that "AI debug interface" -- wouldn't be universal.

Of course, on the flip side, if a universal "AI debug interface" were ever established, perhaps such a thing would eventually suffer from the complexities, over-engineering and bloatedness that plague many "designed-by-committee" standards in today's world.

So, it will be interesting to see what the future holds...

To take an Elon Musk quote and twist it around (basically abuse it! ):

"Proper engineering of future designed-by-committee standards with respect to AI interfaces and protocols is NOT guaranteed -- but excitement is!"

:-) :-)

Anyway, with respect to the main subject/article/authors, it's a very interesting and future-thinking idea what you're doing, you're breaking new ground, and I wish you all of the future success with your company, business, product and product ideas!

Re: Show HN: OpenLLMetry – OpenTelemetry-based observability for LLMs

#18

Post by Areibman »

LLM observability strikes me as an extremely, extremely crowded space. And YC has funded an enormous number of them.

What do you think is the key differentiator between you and everyone else? Is vendor lock-in really that huge of an issue?

[0] https://hegel-ai.com, https://www.vellum.ai/, https://www.parea.ai, http://baserun.ai, https://www.trychatter.ai, https://talc.ai, https://github.com/BerriAI/bettertest, https://langfuse.com

Re: Show HN: OpenLLMetry – OpenTelemetry-based observability for LLMs

#19

Post by nirga »

peter_d_sherman wrote: post #17

Great idea! Observability (AKA, debug/proxy/statistics/logging/visualization layer) -- for LLM's (AKA Chat AI's)... Hmmm, you know, I would love something for ChatGPT (and other AI chatbots) -- where you could open a second tab or window -- and see (and potentially interact with) debug info and statistics from prompts given to that AI in its main input window, in realtime... Sort of like what Unix's STDERR is for pro…

Thanks! Related to what you're saying, I was actually expecting some reactions from devs who'd ask "why is it a separate repo and not part of opentelemetry from day 1?".

And for that my answer would be that I think having a separate repo would allow this to evolve in a more natural way, and faster (whereas OpenTelemetry, given it's massive adoption already, evolves much slower, with committees etc.).

Then, at some point when this is stabilized and useful - we can merge.

Kind of like Tesla's NACS vs. CCS

Re: Show HN: OpenLLMetry – OpenTelemetry-based observability for LLMs

#20

Post by nirga »

Areibman wrote: post #18

LLM observability strikes me as an extremely, extremely crowded space. And YC has funded an enormous number of them. What do you think is the key differentiator between you and everyone else? Is vendor lock-in really that huge of an issue? [0] https://hegel-ai.com , https://www.vellum.ai/ , https://www.parea.ai , http://baserun.ai , https://www.trychatter.ai , https://talc.ai , https://github.com/BerriAI/bettertest ,…

Note that these products aren't the same, even though they all fall under the category of observability - similarly to how you'd use Datadog AND Sentry, although both can be called "observability platforms".

I do think vendor locking is a key differentiator, which some of the reasons why OpenTelemetry succeeded in the first place. I know that my previous company switched to OpenTelemetry for exactly this reason. You get the flexibility of using any platform you'd want (since we're compatible with OpenTelemetry), so it's not vendor-locking you to a specific platform with specific capabilities. Why use any of the ones you mention - maybe Datadog is enough if your use case is simple?

But there are more advantages - you get much more than just observability to the LLM itself - you can see calls to vector DBs, network calls, DB queries, etc. - this can be extremely useful IMO for RAG and autonomous agents for example

Post reply on HN