Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How are generative AI companies monitoring their systems in production?

news.ycombinator.com

Post reply on HN
Print view View original

11–18 of 18 posts

Re: How are generative AI companies monitoring their systems in production?

#11

Post by batshit_beaver »

tikkun wrote: post #7
tikkun wrote: post #6

I've been looking into this question for a bit. [1] Here's my notes on evals -- Things to consider when comparing options: 1) “Types of metrics supported (only NLP metrics, model-graded evals, or both), level of customizability; supports component eval (i.e. single prompts) or pipeline evals (i.e. testing the entire pipeline, all the way from retrieval to post-processing)” 2) “+method of dataset & eval management (co…

Then there's also monitoring. My notes from the monitoring section are below: When needed: “it goes hand-in-hand with eval, as you need to be able to turn bad prod generations into failing eval cases for eng to make pass” Considerations: 1) “Ability to monitor custom metrics (ROUGE [1], Coherence, etc.) and slice-and-dice data - customizability; non-intrusive logging vs proxies, VPC (enterprise-readiness). Plenty of…

I'd suggest looking into WhyLabs. They've got anomaly detection, lightweight SDK, complete data privacy, and ability to ingest custom metrics: https://docs.whylabs.ai/docs/start-here

Re: How are generative AI companies monitoring their systems in production?

#12

Post by tikkun »

batshit_beaver wrote: post #11
tikkun wrote: post #7

Earlier quoted context omitted.

Then there's also monitoring. My notes from the monitoring section are below: When needed: “it goes hand-in-hand with eval, as you need to be able to turn bad prod generations into failing eval cases for eng to make pass” Considerations: 1) “Ability to monitor custom metrics (ROUGE [1], Coherence, etc.) and slice-and-dice data - customizability; non-intrusive logging vs proxies, VPC (enterprise-readiness). Plenty of…

I'd suggest looking into WhyLabs. They've got anomaly detection, lightweight SDK, complete data privacy, and ability to ingest custom metrics: https://docs.whylabs.ai/docs/start-here

Do you have an association with them or just a happy user? Either is fine of course

Re: How are generative AI companies monitoring their systems in production?

#13

Post by batshit_beaver »

tikkun wrote: post #12
batshit_beaver wrote: post #11

Earlier quoted context omitted.

I'd suggest looking into WhyLabs. They've got anomaly detection, lightweight SDK, complete data privacy, and ability to ingest custom metrics: https://docs.whylabs.ai/docs/start-here

Do you have an association with them or just a happy user? Either is fine of course

Former employee, yes. Stumbled across this thread, thought I'd chime in. Didn't realize how many other folks are working on tools for this problem!

Re: How are generative AI companies monitoring their systems in production?

#14

Post by aethelyon »

we built https://klu.ai/ for this

======

outside of us, here's what I see happening

80% of folks aren't building in prod

if you pull apart the 20% that are building, I've seen this from largest to smallest population:

1. most people are not monitoring, followed by 2. home-grown solutions logged into existing observe/analytics platforms, followed by 3. LLMOps tooling like Klu

the 2 cents on the unfortunate truth: I think that many of the AI bolt-on features are living the classic feature lifecycle in that they are launched, no one is monitoring them for improvement, and the feature retention sucks so there's no top-down push to prioritize. the people measuring and improving are exceptional builders regardless of LLMs/RAG.

Re: How are generative AI companies monitoring their systems in production?

#15

Post by ezedv »

It's crucial for AI companies to monitor their systems in production continuously. Not only does this ensure the system's performance and reliability, but it also helps in identifying and addressing any issues or biases that may arise.

Many AI companies use a combination of real-time monitoring, automated alerts, and regular audits to maintain the quality and fairness of their AI systems. It's an ongoing process that plays a vital role in responsible AI development.

In case you have an AI project in mind, feel free to contact us! https://www.ratherlabs.com

Re: How are generative AI companies monitoring their systems in production?

#16

Post by joshreini »

tikkun wrote: post #6

I've been looking into this question for a bit. [1] Here's my notes on evals -- Things to consider when comparing options: 1) “Types of metrics supported (only NLP metrics, model-graded evals, or both), level of customizability; supports component eval (i.e. single prompts) or pipeline evals (i.e. testing the entire pipeline, all the way from retrieval to post-processing)” 2) “+method of dataset & eval management (co…

Bias up front: I'm a TruLens developer

------

We've built all of these considerations:

1) Support for standard metrics like BLEU, ROUGE, BERT similarity, and model-graded evals; we serialize the entire LLM app call so you can test anything that happens in it (context chunks, tool calls/inputs, etc.)

2) Bc we serialize the whole record, you can use this tracing to debug failing evals. We also have chain-of-thought reasoning evals that can explain why an eval failed. Last - there's a streamlit UI you can launch (tru.run_dashboard) that'll run locally

3) Hallucination is probably the biggest problem we solve for. To do evals for hallucination, we typically see our users use a combination of groundedness (does the context support the LLM response) and context relevance (is the retrieved context relevant to the query). There's also a bunch more for the evaluations you mentioned (moderation models, sentiment, usefulness, etc.) and it's pretty easy to add custom evals.

Also - my hot take is that gpt-3.5 is good enough for evals (sometimes better) than gpt-4 if you give the LLM enough instructions on how to do the eval.

website: https://www.trulens.org/ github: https://github.com/truera/trulens

Re: How are generative AI companies monitoring their systems in production?

#17

Post by joshreini »

sirspacey wrote: post #2

Also very interested in this question. We are looking at Truera for observability.

Hey @sirspacey - I'm a TruLens/TruEra dev. If we're not in contact already, feel free to shoot me an email or join our slack community if you need assistance or want to talk through how to best leverage TruEra for this.

email: josh.reini@truera.com slack: https://communityinviter.com/apps/aiqualityforum/josh

Re: How are generative AI companies monitoring their systems in production?

#18

Post by sourabh03agr »

tikkun wrote: post #6

I've been looking into this question for a bit. [1] Here's my notes on evals -- Things to consider when comparing options: 1) “Types of metrics supported (only NLP metrics, model-graded evals, or both), level of customizability; supports component eval (i.e. single prompts) or pipeline evals (i.e. testing the entire pipeline, all the way from retrieval to post-processing)” 2) “+method of dataset & eval management (co…

You can add UpTrain to the list. We are building an open-source LLM evaluation tool with pre-built evaluations such as factual accuracy, retrieval quality, response completeness, tonality, etc. as well as easily extendable framework which allows LLM developers to define their custom evaluations by chaining individual operators. Checkout our demo here: https://demo.uptrain.ai/evals_demo/
Post reply on HN