Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Debugging in the Multiverse

antithesis.com

Post reply on HN
Print view View original

11–20 of 62 posts

Re: Debugging in the Multiverse

#11

Post by wwilson »

ripped_britches wrote: post #9

How do you handle side effects that interact with third party systems? In my own tests, I use network request mocks. Do you need to provide a test mode flag to indicate that mocks should be used?

Any third party service does need to be mocked or stubbed out. We have a partnership with Localstack that lets us provide very polished AWS mocks that require zero configuration on your part (https://antithesis.com/docs/using_antithesis/environment.htm...).

If you need something else, reach out and ask us about it, because we have a few of them in the pipeline.

Re: Debugging in the Multiverse

#12

Post by abeppu »

wwilson wrote: post #7
yellow_lead wrote: post #5

Earlier quoted context omitted.

> Let’s get more concrete. Let’s use this to solve a real problem. My server has crashed and its process has exited! No worries, I’ll just rewind time, attach a debugger to the process, and set a breakpoint or capture a thread dump: Is this kind of stuff only possible in an Antithesis Environment?

Yes, unfortunately we have not figured out how to rewind time in the real world yet. When we do, there are a lot of choices I'm going to revisit...

... but the intro makes it sound like this system is valuable in investigating bugs that occurred in prod systems:

> I’ve been involved in too many production outages and emergencies whose aftermath felt just like that. Eventually all the alerts and alarms get resolved and the error rates creep back down. And then what? Cordon the servers off with yellow police tape? The bug that caused the outage is there in your code somewhere, but it may have taken some outrageously specific circumstances to trigger it.

So practically, if a production outage (where I think "production" means it cannot be in a simulated environment, since the customers you're serving are real) is caused by very specific circumstances, and your production system records some, but not every attribute of its inputs and state ... how does one make use of antithesis? Concretely, when you have a fully-deterministic system that can help your investigation, but you have only a partial view of the conditions that caused the bug ... how do you proceed?

I feel like this post is over-promising but perhaps there's something I just don't understand since I've never worked with a tool set like this.

Re: Debugging in the Multiverse

#13

Post by casept »

Does anything like the Antithesis hypervisor exist as open source?

The closest I've seen is Qemu record/replay, but that's very slow (no KVM acceleration, no multicore), and broken in current Qemu versions (replayed system just gets stuck).

Re: Debugging in the Multiverse

#14

Post by mattgreenrocks »

It is really interesting to me that this sort of thing didn’t come from programming language folks like I’d expect. You’d think PLs are in the absolute perfect spot to implement things, because they define the semantics and runtime. And there are a few PLs who have time-travel demos, but they’ve never really been seen as more than a cool tech demo.

Perhaps the language is too small a vantage point to really get into what’s happening when debugging.

Re: Debugging in the Multiverse

#15

Post by barumrho »

mattgreenrocks wrote: post #14

It is really interesting to me that this sort of thing didn’t come from programming language folks like I’d expect. You’d think PLs are in the absolute perfect spot to implement things, because they define the semantics and runtime. And there are a few PLs who have time-travel demos, but they’ve never really been seen as more than a cool tech demo. Perhaps the language is too small a vantage point to really get into…

Elm debugger did something like this, but it's much more limited in scope.

Re: Debugging in the Multiverse

#16

Post by slippy »

wwilson wrote: post #8
abeppu wrote: post #6

Earlier quoted context omitted.

The intro mentions that ordinarily, we have to pay a high upfront cost to record info that we might need to debug later. > When we succeed at this, we collect huge volumes of logs “just in case” they provide some crucial clue, incurring equally huge storage costs. The 'packets from the past' section says we can just retroactively decide what we should have recorded. Doesn't that mean we're effectively recording every…

You're right that if you tried to do something like this using record/replay, you would pay an enormous cost. Antithesis does not use record/replay, but rather a deterministic hypervisor ( https://antithesis.com/blog/deterministic_hypervisor/ ). So all we have to remember is the set of inputs/changes to entropy that got us somewhere, not the result of every system operation.

The classic time space tradeoff question: If I run Antithesis for X time, say 4 hours, do you take periodic snapshot / deltas of state so that I don't have to re-run the capture for O(4 hours) again, from scratch just to go back 5 seconds?

Re: Debugging in the Multiverse

#17

Post by wwilson »

slippy wrote: post #16
wwilson wrote: post #8

Earlier quoted context omitted.

You're right that if you tried to do something like this using record/replay, you would pay an enormous cost. Antithesis does not use record/replay, but rather a deterministic hypervisor ( https://antithesis.com/blog/deterministic_hypervisor/ ). So all we have to remember is the set of inputs/changes to entropy that got us somewhere, not the result of every system operation.

The classic time space tradeoff question: If I run Antithesis for X time, say 4 hours, do you take periodic snapshot / deltas of state so that I don't have to re-run the capture for O(4 hours) again, from scratch just to go back 5 seconds?

Yes! See Alex's talk here: https://www.youtube.com/watch?v=0E6GBg13P60

In fact, we just made a radical upgrade to this functionality. Expect a blog post about that soon.

Re: Debugging in the Multiverse

#18

Post by 1970-01-01 »

>Seems obvious enough. But maybe, just maybe, the brake lines were cut by somebody who wanted the driver dead. Or what if he was drugged? Can we distinguish that scenario from him being sleepy?

If this is prod, your job is going to be finding what combination of these things caused it this time.

Re: Debugging in the Multiverse

#20

Post by spencerflem »

casept wrote: post #13

Does anything like the Antithesis hypervisor exist as open source? The closest I've seen is Qemu record/replay, but that's very slow (no KVM acceleration, no multicore), and broken in current Qemu versions (replayed system just gets stuck).

There's languages that support time travel debugging, like RR for GDB, or smalltalk, but no open source system wide thing like Antithesis that I know of yet.
Post reply on HN