This looks neat. Will give it a try.
Launch HN: HyperProbe (YC S26) – Agents that do read-only debugging in prod
41–50 of 60 posts
Re: Launch HN: HyperProbe (YC S26) – Agents that do read-only debugging in prod
#42This looks neat. Will give it a try today.
Re: Launch HN: HyperProbe (YC S26) – Agents that do read-only debugging in prod
#43Two things I would want to know before pointing this at a hot service: (1) the overhead budget — when a probe lands on a hot path, is capture sampled or capped per hit, and what p99 latency delta have you measured under load? (2) failure isolation — if probe evaluation itself throws (weird object shape, getter with side effects, huge captured value to serialize), is it contained so it cannot take down the request it…
we have a lot of guardrails (https://docs.hyperprobe.co/how-it-works#built-in-safety-guar...)
if any guardrails fails, we suspend probes till cooldown.
also 1. every probe is bounded by hits/expiry time (whichever comes earlier) 2. hit budgeting happens with a token bucket at a global level, per probe was an overkill (numbers are configurable) 3. we even measure the execution time that probes have when active and suspend if that that takes longer than threshold (again configurable) 4. we even have budgets for the network bandwidth it would take (approximated by the size of payloads) 5. collection itself is bounded by max no of total snapshots we can keep in memory. 6. every snapshot has a size limit as well, every variable has a size limit as well. 7. depth of objects, no of objects, size of lists is capped by default.
latency delta varies by platform under load but is mostly negligible
nodejs: ~7-10ms python: ~4-9ms java: 1-2ms
the main reason for this is guardrails suspending probes, having loosened guardrails will increase this under load
regarding localization of failures.. absolutely we even report the error in the probe snapshot (confirmed by adding side effects in an expression and commenting out the guardrails during testing)
huge payload size doesnt matter.. we limit the objects depth, list length, remove duplicate refs from data etc.. even string length is truncated., but even if it happens, your request would still survive.
also, even if the collector dies or there's a network failure, your service remains unaffected, we just are unable to collect telemetry
we are boring under extreme conditions :)
Re: Launch HN: HyperProbe (YC S26) – Agents that do read-only debugging in prod
#44Read-only in prod is the right constraint. The failure mode I'd most want to hear how you handle isn't a missing signal — it's a confident wrong diagnosis.
Running an autonomous pipeline for eight months, the three incidents that cost me the most days all had the surface error naming the wrong subsystem:
- "x264: malloc of size N failed / incorrect parameters" — I read it as a codec or bad-args bug and went looking there. It was RAM exhaustion. The encoder was the victim, not the cause.
- A 22x slowdown in an LLM step that was indistinguishable from a hang. It was swap: the model no longer fit in RAM, and the page file did the rest.
- A 27-minute "freeze" in a background job. The process was healthy; the pipe was buffering, so nothing appeared until exit.
In all three the logs were complete and the metrics were green. The mistake was in the inference drawn from them — and an agent will produce that wrong inference far faster than I did, with better prose attached to it.
So: does HyperProbe ever return "I don't know — here are two competing hypotheses and the cheapest check that separates them"? The discriminating check is the part I'd pay for. A single confident answer that's wrong is worse than no answer, because it sends a human down a road with the agent's credibility behind it.
Re: Launch HN: HyperProbe (YC S26) – Agents that do read-only debugging in prod
#45Read-only in prod is the right constraint. The failure mode I'd most want to hear how you handle isn't a missing signal — it's a confident wrong diagnosis. Running an autonomous pipeline for eight months, the three incidents that cost me the most days all had the surface error naming the wrong subsystem: - "x264: malloc of size N failed / incorrect parameters" — I read it as a codec or bad-args bug and went looking t…
yes, for example we dont have connectors for k8s yet, so we are blind to memkills triggered due to sidecars.
trying to debug using our tool might even lookup some memleak candidates in your primary container, but there wont be conclusive evidence for it and it would say so.
for in app errors all we do is hypothesize and either prove/disprove that using data from running system.
and whenever we do report something we give have the evidence for it. its not fool proof but just asking does this hypothesis gets proved with this evidence in a subagent mostly does the trick
Re: Launch HN: HyperProbe (YC S26) – Agents that do read-only debugging in prod
#46Congratulations on the launch. Positioning this as an AI-driven debugging layer on top of existing observability tools makes sense, and the read only probes for silent failures feel like a practical way to get runtime evidence without turning every incident into another log and redeploy cycle. Will give this a try for sure!
Re: Launch HN: HyperProbe (YC S26) – Agents that do read-only debugging in prod
#47Spot on and nice complement to the pre-merge half of this problem. Silent logic bugs slips through when a diff "looks right" — what you're building catches them once they're live.
Re: Launch HN: HyperProbe (YC S26) – Agents that do read-only debugging in prod
#48> You "fixed" the incident. You have no idea how. If you don’t know how it broke, and you don’t know how you fixed it, what exactly is it you think you understand about your application?
> If you don’t know how it broke, and you don’t know how you fixed it, what exactly is it you think you understand about your application? what we wanted to convey is that sometimes people confuse "the symptom went away" with "the root cause was fixed" I have seen that a rollback, a quick redeploy, or a temporary drop in tenant load makes the alerts go away and issue is considered resolved. specially true for larger…
> a real example: a dev got OOMed after a release that coincided with a flash sale. he increased memory limits, and containers stopped crashing and it was "fixed". Actualy, a newly introduced internal module had a memory leak. adding RAM just hid the leak until the next traffic spike.
Are you saying that hyperprobe would have in fact caught that issue?
Re: Launch HN: HyperProbe (YC S26) – Agents that do read-only debugging in prod
#49Looks cool, do you have plans to make an open source version for on-premises installation?
Re: Launch HN: HyperProbe (YC S26) – Agents that do read-only debugging in prod
#50Congratulations on the launch. I love the idea and the execution.
When I looked into this a while back I explored using ptrace() to add breakpoints and even add functions at specific line numbers. But ptrace is so slow, and it doesn't work with bytecode-in-VM setups.
What were some of the requirements you guys had when building HyperProbe? I can see low latency was one.