Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Debugging in the Multiverse

antithesis.com

Post reply on HN
Print view View original

41–50 of 62 posts

Re: Debugging in the Multiverse

#41

Post by jackschu »

abeppu wrote: post #12
wwilson wrote: post #7

Earlier quoted context omitted.

Yes, unfortunately we have not figured out how to rewind time in the real world yet. When we do, there are a lot of choices I'm going to revisit...

... but the intro makes it sound like this system is valuable in investigating bugs that occurred in prod systems: > I’ve been involved in too many production outages and emergencies whose aftermath felt just like that. Eventually all the alerts and alarms get resolved and the error rates creep back down. And then what? Cordon the servers off with yellow police tape? The bug that caused the outage is there in your co…

(I work at Antithesis)

I think you're right that the framing leans towards providing value in prod issues, but we left out how we provide value there. I think you're also right that we're just used to experiencing the value here, but it needs some explanation.

Basically this is where guided, tree-based fuzzing comes in. If something in the real world is caused by very specific circumstances, we're well positions to have also generated those specific circumstances. This is thanks to parallelism, intelligent exploration, fault injection, our ability to revisit interesting states in the past with fast snapshots, etc.

We've had some super notable instances of a customer finds a bug in prod, recalls its that weird bug they've been ignoring that we surfaced a month ago, and then uses this approach to debug.

The best docs on this are probably here: https://antithesis.com/docs/introduction/how_antithesis_work...

Re: Debugging in the Multiverse

#42

Post by bobafett-9902 »

quickgist wrote: post #29

I've enjoyed reading many of the blog posts by Antithesis, really cool work. I don't really see a fit for the automated testing product in our stack at the moment, but I would love to use a time traveling hypervisor that I can hop into whenever I'd like. Currently, it seems your pricing is pretty focused on the automated testing service. Do you have pricing or plans that offer just the deterministic dev environment?

(antithesis employee here) We don't currently just offer the deterministic dev environment, but we do offer extended 30 day demos for prospects interested in trying out the tech and seeing how it works. If you're interested contact us directly! contact@antithesis.com

Re: Debugging in the Multiverse

#43

Post by vlovich123 »

mattgreenrocks wrote: post #14

It is really interesting to me that this sort of thing didn’t come from programming language folks like I’d expect. You’d think PLs are in the absolute perfect spot to implement things, because they define the semantics and runtime. And there are a few PLs who have time-travel demos, but they’ve never really been seen as more than a cool tech demo. Perhaps the language is too small a vantage point to really get into…

> Perhaps the language is too small a vantage point to really get into what’s happening when debugging

A little bit. The big thing that others are missing is that it's basically impossible for a PL to accomplish this. Antithesis is basically recording all the state including I/O, network I/O, all RNGS (including the OS) and the big one which everyone has trouble with which is time. So basically you don't need to set up your code and how it interfaces with its environment to be deterministic - you can run within a deterministic container instead which flips the problem on its head and makes it much easier. I'm sure there are tradeoffs. A noteable one is how expensive and slow this approach is vs making your code deterministic. But given how basically no one bothers to make their code deterministic and this is a drop-in solution for scenarios like that, it's really worth it. Additionally, unlike approaches like rr which offer similar capabilities, this is even more generic & not dependent on adding support for every OS interface (e.g. rr doesn't support io_uring yet but I believe antithesis would since it's running at the VM level)

Re: Debugging in the Multiverse

#44

Post by vlovich123 »

munificent wrote: post #34
mattgreenrocks wrote: post #14

It is really interesting to me that this sort of thing didn’t come from programming language folks like I’d expect. You’d think PLs are in the absolute perfect spot to implement things, because they define the semantics and runtime. And there are a few PLs who have time-travel demos, but they’ve never really been seen as more than a cool tech demo. Perhaps the language is too small a vantage point to really get into…

I know time-travel debugging is very very close to Gilad Bracha's heart and something he was really hoping would make its way into Dart. I don't know to what degree this is true for other language teams but one thing I've observed is that language designers, compiler people, VM people, and IDE/debugger people have more distinct cultures than you might expect. That can make it hard to ship features that cut across tho…

There's reverse debugging and then there's what antithesis does which is a deterministic guarantee of the state. So for example, if you rewind, you'll get the exact same disk & network I/O happening across each call. And it supports arbitrary OS operations whereas typically at the PL level you'll be left at the mercy of whichever OS APIs the PL chooses to support for recording (i.e. similar to rr in terms of what it'll be able to do). Often times, PLs don't even bother with recording state across OS calls since they don't actually know what are OS calls vs normal function calls.

Re: Debugging in the Multiverse

#45

Post by vlovich123 »

dzaima wrote: post #32
casept wrote: post #31

Earlier quoted context omitted.

My experience with RR is that the chance of it working without hitting a missing syscall or desync is only about 50%, which is why I want a different solution that doesn't rely on the fragile syscall recording approach.

Huh. In my experience it works nearly flawlessly, certainly far above 50%. And even when there are spurious failures in replaying it's easy enough to just re-replay (though I do wish there was some way to export the current position & checkpoints with instruction-level precision to import in a fresh replay). I suppose it depends massively on the recorded program (most of mine are simple C programs, but also a decent…

Same, I haven't had it have too many problems but it's not perfect & missing support for io_uring is a problem (they'll add it eventually I suspect once someone ponies up the money for it).

Re: Debugging in the Multiverse

#46

Post by jackschu »

mattgreenrocks wrote: post #14

It is really interesting to me that this sort of thing didn’t come from programming language folks like I’d expect. You’d think PLs are in the absolute perfect spot to implement things, because they define the semantics and runtime. And there are a few PLs who have time-travel demos, but they’ve never really been seen as more than a cool tech demo. Perhaps the language is too small a vantage point to really get into…

Yeah it's fun that we get to do this at the hypervisor level. This opens up time-traveling in systems where there's cross-machine or inter-process communication, which really widens what we're able to do.

(I work at Antithesis, if youre interested in chatting more once this thread has gone cold come join discord.gg/antithesis)

Re: Debugging in the Multiverse

#47

Post by grumbelbart »

I was once working in a company producing software / operating systems for smart cards (such as the chips on your credit cards). We developed a simulator for the hardware that logged all changes to registers, memory and other states in a very large ring buffer, allowing us to undo / step backwards through code. With RAM being large, those chips being slow, and some snapshotting, we were usually able to undo back to the reset of the card. That was a game changer regarding debugging the OS.

Re: Debugging in the Multiverse

#48

Post by Terr_ »

emeryberger wrote: post #39

Would love to hear a technical comparison between this and King et al.'s classic paper on Time-Traveling VMs from USENIX ATC 2006: "Debugging operating systems with time-traveling virtual machines" ( https://www.usenix.org/legacy/events/usenix05/tech/general/k... , 505 citations).

Seguing to talks regarding literal time traveling VMs, I'm reminded of Damian Conway's "Temporally Quaquaversal Virtual Nanomachine Programming In Multiple Topologically Connected Quantum-Relativistic Parallel Spacetimes... Made Easy!" presentation.

Essentially, it involves a series of sci-fi concepts, and then showing the kind of program (in modified perl) that someone might write to take advantage of those capabilities.

Re: Debugging in the Multiverse

#49

Post by terpimost »

bluelightning2k wrote: post #37

I know I'm taking the wrong thing from this - but I really struggle to read this site. Something about the contrast and aggro gradients.

Hey, designer here. Thank you for this feedback. Do you prefer dark or light theme usually? And do you find reading text on this background here https://antithesis.com/security/manifesto/ is any easier?
Post reply on HN