Interesting! tbh, we don't have any runbooks and pretty minimal telemetry set up (we're a very small team :), do you have any recommendations on which telemetry service to use to get started? right now, our services run on a combination GCP Cloud Run + Vercel
Launch HN: Relvy (YC F24) – On-call runbooks, automated
21–30 of 35 posts
Re: Launch HN: Relvy (YC F24) – On-call runbooks, automated
#22Interesting! tbh, we don't have any runbooks and pretty minimal telemetry set up (we're a very small team :), do you have any recommendations on which telemetry service to use to get started? right now, our services run on a combination GCP Cloud Run + Vercel
Nice to see you here, Will! I’d generally recommend using open telemetry for instrumentation so that you keep the option of switching between telemetry vendors.
Re: runbooks, yeah even larger teams don’t have good ones to begin with. Relvy helps debug without runbooks as well - it might take longer to explore, but once you are happy with a particular investigation path the AI took, you can save it as a runbook for more deterministic future executions.
Re: Launch HN: Relvy (YC F24) – On-call runbooks, automated
#23[dead]
Re: Launch HN: Relvy (YC F24) – On-call runbooks, automated
#24Interesting!
In my experience using custom harnesses has worked better eg: Stripe etc all did it custom largely because of the sensitive integrations. How would you handle that?
Re: Launch HN: Relvy (YC F24) – On-call runbooks, automated
#25Congratulations! The difference between pure agentic exploration and deterministic steps is spot on. Runbooks give ops more confidence on the data exploration and save time/context.
Curious how much savings do you observe from using runbook versus purely let Claude do the planning at first. Also how the runbooks can self heal if results from some steps in the middle are not expected.
Re: Launch HN: Relvy (YC F24) – On-call runbooks, automated
#26Interesting! In my experience using custom harnesses has worked better eg: Stripe etc all did it custom largely because of the sensitive integrations. How would you handle that?
Do you mean how we connect to internal data? Today, you can connect any API endpoint to Relvy, so if you have internal business data / dashboards that you look at while debugging, Relvy can do the same if there's an API for it.
Most of our deployments are self-hosted, in which case the data stays locally (your chosen LLM provider exempted), if that's what you are asking.
Re: Launch HN: Relvy (YC F24) – On-call runbooks, automated
#27Interesting! In my experience using custom harnesses has worked better eg: Stripe etc all did it custom largely because of the sensitive integrations. How would you handle that?
[deleted]
Re: Launch HN: Relvy (YC F24) – On-call runbooks, automated
#28Congratulations! The difference between pure agentic exploration and deterministic steps is spot on. Runbooks give ops more confidence on the data exploration and save time/context. Curious how much savings do you observe from using runbook versus purely let Claude do the planning at first. Also how the runbooks can self heal if results from some steps in the middle are not expected.
>> how the runbooks can self heal if results from some steps in the middle are not expected.
Yeah this is a very interesting angle. Our primary mechanism here is via agent created auto-memories today. The agent keeps track of the most useful steps, and more importantly, dead end steps as it executes runbooks. We think this offers a great bridge to suggest runbook updates and keep them current.
>> Curious how much savings do you observe from using runbook versus purely let Claude do the planning at first.
Really depends on runbook quality, so I don't have a straightforward answer. Of course, it's faster and cheaper if you have well defined steps in your runbooks. As an example, `check logs for service frontend, faceted by host_name`, vs. `check logs`. Agent does more exploration in the latter case.
We wrote about the LLM costs of investigating production alerts more generally here, in case helpful: https://relvy.ai/blog/llm-cost-of-ai-sre-investigating-produ...
Re: Launch HN: Relvy (YC F24) – On-call runbooks, automated
#29Interesting! In my experience using custom harnesses has worked better eg: Stripe etc all did it custom largely because of the sensitive integrations. How would you handle that?
Re: custom harnesses, imo maintaining them can be time consuming especially when things are changing very fast with AI. Bringing up a prototype is easy but a robust harness that handles the edge cases needs time and effort.
Re: Launch HN: Relvy (YC F24) – On-call runbooks, automated
#30Congratulations! The difference between pure agentic exploration and deterministic steps is spot on. Runbooks give ops more confidence on the data exploration and save time/context. Curious how much savings do you observe from using runbook versus purely let Claude do the planning at first. Also how the runbooks can self heal if results from some steps in the middle are not expected.
Re: savings - it depends on the use case. For example, one of our users set up a small runbook to run a group-by-IP query for high-throughput alerts, since that was their most common first response to those alerts. That alone cuts out a couple of minutes of exploration per incident and removes the variability of the agent deciding what data to investigate and how to slice it.
In our experience, runbooks provide a consistent, fast, and reliable way of investigating incidents (or ruling out common causes). In their absence, the AI does its usual open-ended exploration.