Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Parity (YC S24) – AI for on-call engineers working with Kubernetes

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 78 posts

Re: Launch HN: Parity (YC S24) – AI for on-call engineers working with Kubernetes

#41

Post by cortesoft »

upon_drumhead wrote: post #9
Atotalnoob wrote: post #2

It would be kind of interesting if, based on an engineer accepting the suggestion, parity generated a new run book. This would allow repeated issues to be well documented. On iOS Firefox, when clicking “pricing” on the menu, it scrolls to the proper location, but does not close the menu. Closing the menu causes it to jump to the top of the page. Super annoying.

If an issue can be automatically detected and remediated, do you really need a runbook? That space has to be huge. I don't see a purpose for documenting it. That said, a tool that runs through existing runbooks and improves them or suggests new ones would be extremely useful IMHO.

> I don't see a purpose for documenting it.

Because when it goes wrong you will want to know what it did. When you discover something new, you are going to want to be able to change the runbook. New employees are going to want to learn how things work from the runbook.

Why WOULDN'T you want to document what it is doing? I would never trust an AI that didn't tell me what it was doing and why.

Re: Launch HN: Parity (YC S24) – AI for on-call engineers working with Kubernetes

#42

Post by bicx »

threeseed wrote: post #23
bicx wrote: post #21

Earlier quoted context omitted.

Getting tired of seeing this concept of practically guaranteed hallucinations from any LLM used in production. I've used LLMs for various tasks, and if you tune your system correctly, it can be very reliable. It's just not always plug-and-play reliability. You need to set up your fine-tuning and prompts and then test well for consistent results.

> it can be very reliable You need to quantify this. With actual numbers. I am getting very tired of seeing everyone pushing LLMs and being disingenuous about exactly how often it is getting things wrong. And what the impact of that is. There is a reason that AI is not taking off in the enterprise and that is because people who take their job seriously are getting tired too.

I understand your sentiment, but I also don't think it's fair to say people are being disingenuous. I don't work for an AI company, I just use it with decent results.

My last project needed a way to tag certain types of business activity indicated in 17,000 user reviews. I wrote a prompt with 5 different tags along with descriptions for each business activity, took a few-shot approach by defining 8 different examples and how I would tag them, and then ran the tagging prompt on batches of indexed reviews, giving it 100 reviews per batch. I did a random sampling of about 200 items, and the success rate was roughly 89%. I could have improved it by doing more iterations, and possibly fine-tuning if I felt that it was worth it.

In every run, it generated matching results for the input JSON in a corresponding output JSON with no errors in any run.

That's the only example I have numbers on off the top of my head.

EDIT: This was using `gpt-4o-2024-05-13`

Re: Launch HN: Parity (YC S24) – AI for on-call engineers working with Kubernetes

#43

Post by jtsaw »

threeseed wrote: post #36
jtsaw wrote: post #27

Earlier quoted context omitted.

one thing we're experimenting to help with the hallucinations/error rate issue is using a committee framework where we take a majority vote. If the error rate of 1 expert is 5%, then for a committee of 10 experts, the probability a majority of the committee errors is around 0.00276% (binomial distribution with p=0.05). For 10 steps, this would be an error rate of 0.0276%

Pretty bad maths there. Those committee members are not independent. They are highly correlated even amongst LLMs from different vendors.

I'm not sure they are highly correlated. A committee uses the same LLM with the same input context to generate different outputs. Given the same context LLMs should produce the same next token output distribution (assuming fixed model parameters, temperature, etc). So, while tokens in a specific output are highly correlated, complete outputs should be independent since they are generated independently from the same distribution. You are right they are not iid but the calculation was just a simplification.

Re: Launch HN: Parity (YC S24) – AI for on-call engineers working with Kubernetes

#44

Post by solatic »

wilson090 wrote: post #13
stackskipton wrote: post #6

Azure Kubernetes Wrangler (SRE) here, before I turn some LLM loose on my cluster, I need to know what it supports, how it supports it and how I can integrate into my workflow. Videos show CrashLoopBackOff pod and analyzing logs. This works if Pod is writing to stdout but I've got some stuff doing straight to ElasticSearch. Does LLM speak Elastic Search? How about Log Files in the Pod? (Don't get me started on that ni…

Your comment brings up a good point (and also one of our big challenges): there is a huge diversity in the tools teams use to setup and operate their infra. Right now our platform only speaks to your cluster directly through kubectl commands. We’ll build other integrations so it can communicate with things like Elastic Search to broaden its context as needed, but we’ll have to be somewhat thoughtful in picking the hi…

> we think SREs will still find it useful

There are two kinds of outages: people being idiots and legit hard-to-track-down bugs. SREs worth their salt don't need help with the former. They may find an AI bot somewhat useful to find root cause quicker, but usually not so valuable as to justify paying the kind of price you would need to charge to make your business viable to VCs. As for the latter, good luck collecting enough training data.

Otherwise, you're selling a self-driving car to executives who want the chauffeur without the salary. Sounds like a great idea, until you think about the tail cases. Then you wish you had a chauffeur (or picked up driving skills yourself).

Maybe you'll find a market, but as an SRE, I wouldn't want to sell it.

Re: Launch HN: Parity (YC S24) – AI for on-call engineers working with Kubernetes

#45

Post by wilson090 »

shmatt wrote: post #40
stackskipton wrote: post #6

Azure Kubernetes Wrangler (SRE) here, before I turn some LLM loose on my cluster, I need to know what it supports, how it supports it and how I can integrate into my workflow. Videos show CrashLoopBackOff pod and analyzing logs. This works if Pod is writing to stdout but I've got some stuff doing straight to ElasticSearch. Does LLM speak Elastic Search? How about Log Files in the Pod? (Don't get me started on that ni…

Im sure this is on their roadmap, but honestly a pre-requisite should be a separate piece of software that analyzes and suggests changes to your error handling. This is a cool proof of concept but almost useless otherwise in a production system I can already feed Claude or ChatGPT my kubectl output pretty easily Error handling and logging that are tailored for consumption of a specific pre trained model, thats where…

That is something we're working on -- good observability is a place where teams usually fall short and often the limiting factor to better incident response. We're working on logging integrations as a first step.

Re: Launch HN: Parity (YC S24) – AI for on-call engineers working with Kubernetes

#47

Post by drawnwren »

nerdjon wrote: post #25
bicx wrote: post #21

Earlier quoted context omitted.

Getting tired of seeing this concept of practically guaranteed hallucinations from any LLM used in production. I've used LLMs for various tasks, and if you tune your system correctly, it can be very reliable. It's just not always plug-and-play reliability. You need to set up your fine-tuning and prompts and then test well for consistent results.

> You need to set up your fine-tuning and prompts and then test well for consistent results. Tell that to Google... Seriously, it is well established that these systems hallucinate. Trying to say otherwise shows you are trying to push something that just is not true. They can be right, yes. But when they are wrong they can be catastrophically wrong. You could be wasting time looking into the wrong problem with someth…

If you're curious what the state of the art in multi-agent is looking like, I really recommend https://thinkwee.top/multiagent_ebook/

Re: Launch HN: Parity (YC S24) – AI for on-call engineers working with Kubernetes

#48

Post by wilson090 »

drawnwren wrote: post #46

This is a great idea. I use claude for my most of my unknown K8s bugs and it's impressive how useful it is (far more than my coding bugs).

Thanks! We've also been impressed with the performance of out-of-the-box LLMs on this use case. I think in part it is because k8s is a significantly more constrained problem-space than coding, and because of that we'll get to a much more complete solution with the existing state of LLMs than we would for a product like a general software engineer agent.

Re: Launch HN: Parity (YC S24) – AI for on-call engineers working with Kubernetes

#49

Post by ronald_petty »

I think this kind of tooling is one positive aspect of integrating LLM tech in certain workflows/pipelines. Tools like k8sgpt are similar in purpose and show a strong potential to be useful. Look forward to seeing how this progresses.

Re: Launch HN: Parity (YC S24) – AI for on-call engineers working with Kubernetes

#50

Post by stackskipton »

shmatt wrote: post #40
stackskipton wrote: post #6

Azure Kubernetes Wrangler (SRE) here, before I turn some LLM loose on my cluster, I need to know what it supports, how it supports it and how I can integrate into my workflow. Videos show CrashLoopBackOff pod and analyzing logs. This works if Pod is writing to stdout but I've got some stuff doing straight to ElasticSearch. Does LLM speak Elastic Search? How about Log Files in the Pod? (Don't get me started on that ni…

Im sure this is on their roadmap, but honestly a pre-requisite should be a separate piece of software that analyzes and suggests changes to your error handling. This is a cool proof of concept but almost useless otherwise in a production system I can already feed Claude or ChatGPT my kubectl output pretty easily Error handling and logging that are tailored for consumption of a specific pre trained model, thats where…

The AI needs to be integrated into Dev IDE. All my logging screaming is terrible decisions made by long ago Devs but getting them fixed now is impossible because they don't want to do it and no one is going to make them.
Post reply on HN