Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Langfuse – Open-source observability and analytics for LLM apps

github.com

Post reply on HN
Print view View original

21–30 of 39 posts

Re: Show HN: Langfuse – Open-source observability and analytics for LLM apps

#21

Post by phillipcarter »

mdeichmann wrote: post #18
phillipcarter wrote: post #13

Congrats on the release! I'm keenly interested in this space, as I believe that Observability is one of the top ways to steer LLMs to be more reliable in production. I noticed your SDKs use tracing concepts! Are there plans to implement OpenTelemetry support?

Thank you so much, fully share your sentiment on this and aligned our domain language to OpenTelemetry. Currently users add lots of metadata and configuration details to the trace by manually instrumenting it using the SDKs (or via Langchain integration). We are thinking about integrating OpenTelemetry, as this would be a step function on making integrations with apps easier. However, hadn't had the time yet to figur…

Makes sense! If you're curious, I added an autoinstrumentation library for openai's python client here: https://github.com/cartermp/opentelemetry-instrument-openai-...

The main challenge I see is that since there's no standard that each LLM has for inputs/outputs (let alone retrieval APIs!) any kind of automatic instrumentation will need to have a bunch of adapters. I suppose LangChain helps here, but even then with so many folks ripping it out for production you're still in the same place.

Happy to collaborate on any design thinking for how to incorporate OTel support!

Re: Show HN: Langfuse – Open-source observability and analytics for LLM apps

#22

Post by mdeichmann »

phillipcarter wrote: post #21
mdeichmann wrote: post #18

Earlier quoted context omitted.

Thank you so much, fully share your sentiment on this and aligned our domain language to OpenTelemetry. Currently users add lots of metadata and configuration details to the trace by manually instrumenting it using the SDKs (or via Langchain integration). We are thinking about integrating OpenTelemetry, as this would be a step function on making integrations with apps easier. However, hadn't had the time yet to figur…

Makes sense! If you're curious, I added an autoinstrumentation library for openai's python client here: https://github.com/cartermp/opentelemetry-instrument-openai-... The main challenge I see is that since there's no standard that each LLM has for inputs/outputs (let alone retrieval APIs!) any kind of automatic instrumentation will need to have a bunch of adapters. I suppose LangChain helps here, but even then with…

Yes, we were thinking about the lack of standards as well. I would be super happy to have a design discussion around the topic, i will reach out to you.

Re: Show HN: Langfuse – Open-source observability and analytics for LLM apps

#23

Post by v3np »

Cool stuff and congrats on the Show HN! Out of curiosity, at what point do you see teams usually adopting something like langfuse? In regular development, you sometimes even have test-driven development - I imagine this doesn't really apply for LLMs. Do you see this changing over time as the process of building LLM apps becomes more mature?

Re: Show HN: Langfuse – Open-source observability and analytics for LLM apps

#24

Post by mdeichmann »

addisonj wrote: post #20

Congrats on the launch! I have quite a few years of observability experience behind me and hand't really considered some of the unique aspects that LLMs bring into the picture. Here are a few thoughts, responses to your questions, and feedback items * Generally, I think you do a good job of having a clear, concise story and value proposition that is fairly early in a market where the number of people hitting these pr…

This reads like a book, thank you so much for putting this together!

> About value prop: Thanks for the feedback! We are already trying to be as vocal about it as possible by writing great docs etc. but can probably do better.

> PLG & OSS: thanks for the hint, we will be careful around managing deployments within customer VPCs.

> Pricing: Currently picked storage as the first metric to price on as this varies a lot across users. Some use langfuse to track complex embedding processes with a lot of context, others just simple chat messages with relatively low-context, low-value events.

> OTel: We looked into it but did not go into all the details. We wanted to have a product out there fast and liked the experience of e.g. Posthog SDKs. I might reach out to you concerning this topic after investing more time on it. Thanks for the offer!

> OLAP: Agree, i also learned to tackle scaling issues once they appear and so far we are good. Interesting that Supabase has no horizontal scaling. This would be one of the main reasons to use it IMO.

Re: Show HN: Langfuse – Open-source observability and analytics for LLM apps

#25

Post by mdeichmann »

v3np wrote: post #23

Cool stuff and congrats on the Show HN! Out of curiosity, at what point do you see teams usually adopting something like langfuse? In regular development, you sometimes even have test-driven development - I imagine this doesn't really apply for LLMs. Do you see this changing over time as the process of building LLM apps becomes more mature?

Thanks a lot! We see teams adopt Langfuse quite early already. Say you have one or two engineers working on a rather complex LLM feature, they look for a solution like Langfuse already in a test environment before going to production. The majority observes their LLM features in production though. We dont see test-driven development as much but we do think that model and rule based eval will become more important in the future and CIs will only pass if a certain score was achieved.

Re: Show HN: Langfuse – Open-source observability and analytics for LLM apps

#26

Post by hrpnk »

addisonj wrote: post #20

Congrats on the launch! I have quite a few years of observability experience behind me and hand't really considered some of the unique aspects that LLMs bring into the picture. Here are a few thoughts, responses to your questions, and feedback items * Generally, I think you do a good job of having a clear, concise story and value proposition that is fairly early in a market where the number of people hitting these pr…

+1 on the OTel mention. Having telemetry in place in a system, one would typically implement a single behavioral tracking SDK on top. Adding yet another SDK for LLMs is a hard ask given how specific the implementation will be. Backing back on a standard you offer value-added insights on top.

On the other hand, if you target just the applications that implement an API behind an LLM, you will have customers expecting value-added services on top of telemetry, like prompt optimization, classification, result caching, etc.

Your choice which direction and target group you will focus on first.

Re: Show HN: Langfuse – Open-source observability and analytics for LLM apps

#27

Post by anirudhrx »

Congrats on the launch! This is really cool. Would love to see OTel integration in the future. I'm curious if this might eventually work with request-context based routing, i.e. being able to use the propagated metadata between layers to dynamically test different versions of the stack, replay requests / route to specific underlying implementation versions at different levels of the stack.

Re: Show HN: Langfuse – Open-source observability and analytics for LLM apps

#28

Post by marcklingen »

anirudhrx wrote: post #27

Congrats on the launch! This is really cool. Would love to see OTel integration in the future. I'm curious if this might eventually work with request-context based routing, i.e. being able to use the propagated metadata between layers to dynamically test different versions of the stack, replay requests / route to specific underlying implementation versions at different levels of the stack.

Thx. Currently looking deeply into how we can make it as dynamic as possible to help people experiment with configurations and test on production samples. What do you exactly refer to with levels/layer in stack? See multiple meanings that would all make sense to me and would love to learn more

Re: Show HN: Langfuse – Open-source observability and analytics for LLM apps

#29

Post by anirudhrx »

marcklingen wrote: post #28
anirudhrx wrote: post #27

Congrats on the launch! This is really cool. Would love to see OTel integration in the future. I'm curious if this might eventually work with request-context based routing, i.e. being able to use the propagated metadata between layers to dynamically test different versions of the stack, replay requests / route to specific underlying implementation versions at different levels of the stack.

Thx. Currently looking deeply into how we can make it as dynamic as possible to help people experiment with configurations and test on production samples. What do you exactly refer to with levels/layer in stack? See multiple meanings that would all make sense to me and would love to learn more

Sorry if I'm not using the right terminology here. By layers, I'm talking about the different places from where you're emitting spans - like in your example, the UI, prompt-embedding, vector-store and context-encoding. What I was curious about was two-fold - one whether there are use-cases that make sense to have experimentation of the kind that allows a user to swap out a particular step in that stack - and second about making use of context propagation like baggage enables in OTel.

Re: Show HN: Langfuse – Open-source observability and analytics for LLM apps

#30

Post by marcklingen »

anirudhrx wrote: post #29
marcklingen wrote: post #28

Earlier quoted context omitted.

Thx. Currently looking deeply into how we can make it as dynamic as possible to help people experiment with configurations and test on production samples. What do you exactly refer to with levels/layer in stack? See multiple meanings that would all make sense to me and would love to learn more

Sorry if I'm not using the right terminology here. By layers, I'm talking about the different places from where you're emitting spans - like in your example, the UI, prompt-embedding, vector-store and context-encoding. What I was curious about was two-fold - one whether there are use-cases that make sense to have experimentation of the kind that allows a user to swap out a particular step in that stack - and second a…

Got it, appreciate the clarification.

(1) yes, that's what many of our users are doing while swapping might refer to changing model parameters, prompts, embedding stores, embedding parameters, or splitting a "layer" into multiple (parallel) steps. Alternatively when experimenting on a higher "layer", we often see wildly different implementations that are experimented with over time. We added version and release tagging which to build dashboards on the impact of these experiments on quality/latency/cost (currently in alpha, we will release it soon).

(2) I see how metadata propagation could be much more elegant via OTel. Currently the metadata fields on traces/span and filtering based on them kind of does the job but I see how it could be much easier to ingest and use. Thank you!

Post reply on HN