Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Parity (YC S24) – AI for on-call engineers working with Kubernetes

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 78 posts

Re: Launch HN: Parity (YC S24) – AI for on-call engineers working with Kubernetes

#31

Post by mplewis »

bicx wrote: post #21
threeseed wrote: post #19

> This agent is a combination of separate LLM agents each responsible for a single step of the runbook Someone needs to explain to me how this is expected to work. Percentage of Hallucinations/Errors x Steps in Runbook = Total Errors 0.05 x 10 = 0.5 = 50%

Getting tired of seeing this concept of practically guaranteed hallucinations from any LLM used in production. I've used LLMs for various tasks, and if you tune your system correctly, it can be very reliable. It's just not always plug-and-play reliability. You need to set up your fine-tuning and prompts and then test well for consistent results.

Every LLM conversation is guaranteed to contain some level of hallucination. You will never get the percentage down to zero.

Re: Launch HN: Parity (YC S24) – AI for on-call engineers working with Kubernetes

#32

Post by wilson090 »

manveerc wrote: post #29

Congratulations on the launch! I'm curious—how is what you're building different from other AI SRE solutions out there, like Cleric, Onegrep, Resolve, Beeps, and others?

Thanks! Hard to make a comparison to Cleric since their site doesn't really have any features or a demo. Onegrep is a fellow YC company, and we love what they're building! They seem to be more focused on workflows and pulling together context (also a very important problem in the space), we we've put more of a focus on root causing infra issues specifically. Resolve seems to come from the same category as Temporal, which are more traditional automation platforms. These end up being somewhat rigid tools in that you have to very explicitly define each step and they require a certain level of CI/CD or monitoring sophistication to be useful. Using LLMs allows us to relax these requirements and follow workflows like an actual engineer would.

I haven't heard of Beeps and can't find it, could you share the URL?

Re: Launch HN: Parity (YC S24) – AI for on-call engineers working with Kubernetes

#33

Post by jtsaw »

nerdjon wrote: post #28
wilson090 wrote: post #22

Earlier quoted context omitted.

We separated out the runbook such that each step is a separate LLM in the agent. Between each step, there's sort of a "supervisor" that ensures that the step was completed correctly, and then routes to another step based on the results. So in reality, a single step failing requires two hallucinations. Hallucinations are also not a fixed percentage across all calls -- you can make them less likely by maintaining focus…

That second step hallucinating is far more likely when you are feeding it incorrect information from the first hallucination. LLM's are very easy to manipulate. At one point with a system prompt telling Claude it was OpenAI, I was able to ask what its model is and it would confidently tell me it was OpenAI. Garbage data in, garbage data out. Admittedly that is an extreme case, but you're giving that second prompt wro…

yea. We're definitely concerned about hallucinations and are using a variety of techniques to try and mitigate it (there's some existing discussion here, but using committees and sub-agents responsible for smaller tasks has helped).

What's helped the most, though, is using cluster information to back up decision making. That way we know the data it's considering isn't garbage, and the outputs are backed up by actual data.

Re: Launch HN: Parity (YC S24) – AI for on-call engineers working with Kubernetes

#34

Post by manveerc »

wilson090 wrote: post #32
manveerc wrote: post #29

Congratulations on the launch! I'm curious—how is what you're building different from other AI SRE solutions out there, like Cleric, Onegrep, Resolve, Beeps, and others?

Thanks! Hard to make a comparison to Cleric since their site doesn't really have any features or a demo. Onegrep is a fellow YC company, and we love what they're building! They seem to be more focused on workflows and pulling together context (also a very important problem in the space), we we've put more of a focus on root causing infra issues specifically. Resolve seems to come from the same category as Temporal, w…

> They seem to be more focused on workflows and pulling together context (also a very important problem in the space), we we've put more of a focus on root causing infra issues specifically.

So just to clarify, are you saying that Parity is focused on infrastructure issues, while something like Onegrep addresses the broader problem by providing context?

> I haven't heard of Beeps and can't find it, could you share the URL?

https://www.beeps.co/

Re: Launch HN: Parity (YC S24) – AI for on-call engineers working with Kubernetes

#36

Post by threeseed »

jtsaw wrote: post #27
threeseed wrote: post #24

Earlier quoted context omitted.

And what is your average error rate per runbook step.

one thing we're experimenting to help with the hallucinations/error rate issue is using a committee framework where we take a majority vote. If the error rate of 1 expert is 5%, then for a committee of 10 experts, the probability a majority of the committee errors is around 0.00276% (binomial distribution with p=0.05). For 10 steps, this would be an error rate of 0.0276%

Pretty bad maths there. Those committee members are not independent.

They are highly correlated even amongst LLMs from different vendors.

Re: Launch HN: Parity (YC S24) – AI for on-call engineers working with Kubernetes

#37

Post by wilson090 »

manveerc wrote: post #34
wilson090 wrote: post #32

Earlier quoted context omitted.

Thanks! Hard to make a comparison to Cleric since their site doesn't really have any features or a demo. Onegrep is a fellow YC company, and we love what they're building! They seem to be more focused on workflows and pulling together context (also a very important problem in the space), we we've put more of a focus on root causing infra issues specifically. Resolve seems to come from the same category as Temporal, w…

> They seem to be more focused on workflows and pulling together context (also a very important problem in the space), we we've put more of a focus on root causing infra issues specifically. So just to clarify, are you saying that Parity is focused on infrastructure issues, while something like Onegrep addresses the broader problem by providing context? > I haven't heard of Beeps and can't find it, could you share th…

Yes, my understanding is that Onegrep is meant to provide context from your documentation and past incidents, which can be helpful when trying to solve an alert. We're focused on root-causing underlying infrastructure issues by actually looking into the logs/configurations/metrics.

Ah I actually did see beeps a while back. I haven't tried their product, but they seem to be similar to rootly/Onegrep in that they're working on on-call management/post-mortems

Re: Launch HN: Parity (YC S24) – AI for on-call engineers working with Kubernetes

#39

Post by solatic »

stackskipton wrote: post #6

Azure Kubernetes Wrangler (SRE) here, before I turn some LLM loose on my cluster, I need to know what it supports, how it supports it and how I can integrate into my workflow. Videos show CrashLoopBackOff pod and analyzing logs. This works if Pod is writing to stdout but I've got some stuff doing straight to ElasticSearch. Does LLM speak Elastic Search? How about Log Files in the Pod? (Don't get me started on that ni…

I basically want to +1 this. OP isn't selling to any place that is already spending six figures on SRE salaries. Actual competitors are companies like Komodor and Robusta who sell "we know Kubernetes better than you" solutions to companies that don't want to spend money on SRE salaries. Companies in this situation should just seriously reconsider hosting on Kubernetes and go back to higher-level managed services like ECS/Fargate, Fly/Railway, etc.

Re: Launch HN: Parity (YC S24) – AI for on-call engineers working with Kubernetes

#40

Post by shmatt »

stackskipton wrote: post #6

Azure Kubernetes Wrangler (SRE) here, before I turn some LLM loose on my cluster, I need to know what it supports, how it supports it and how I can integrate into my workflow. Videos show CrashLoopBackOff pod and analyzing logs. This works if Pod is writing to stdout but I've got some stuff doing straight to ElasticSearch. Does LLM speak Elastic Search? How about Log Files in the Pod? (Don't get me started on that ni…

Im sure this is on their roadmap, but honestly a pre-requisite should be a separate piece of software that analyzes and suggests changes to your error handling.

This is a cool proof of concept but almost useless otherwise in a production system

I can already feed Claude or ChatGPT my kubectl output pretty easily

Error handling and logging that are tailored for consumption of a specific pre trained model, thats where this will be ground breaking

Post reply on HN