Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is something bugging you?

antithesis.com

Post reply on HN
Print view View original

281–290 of 438 posts

Re: Is something bugging you?

#281

Post by pokler »

wwilson wrote: post #143
randomdata wrote: post #134

Earlier quoted context omitted.

> most software is non-deterministic Doesn't Antithesis rely on the fact that software is always deterministic? Reproducibility appears to be its top selling feature – something that wouldn't be possible if software were non-deterministic.

We can force any* software to be deterministic. * Offer only good for x86-64 software that runs on Linux whose dependencies you can install locally or mock. The first two restrictions we will probably relax someday.

That point about dependencies -- how well does this play or easy to integrate with a build system like Bazel or Buck?

Re: Is something bugging you?

#282

Post by acemarke »

vinnymac wrote: post #17

I wonder if they are working on a time travel debugger. If it is truly deterministic presumably you could visit any point in time after a record is made and replay it.

Exactly - that's what we've already built for web development at https://replay.io :)

I did a "Learn with Jason" show discussion that covered the concepts of Replay, how to use it, and how it works:

- https://www.learnwithjason.dev/travel-through-time-to-debug-...

Not only is the debugger itself time-traveling, but those time-travel capabilities are exposed by our backend API:

- https://static.replay.io/protocol/

Our entire debugging frontend is built on that API. We've also started to build new advanced features that leverage that API in unique ways, like our React and Redux DevTools integration and "Jump to Code" feature:

- https://blog.replay.io/how-we-rebuilt-react-devtools-with-re...

- https://blog.isquaredsoftware.com/2023/10/presentations-reac...

- https://github.com/Replayio/Protocol-Examples

Re: Is something bugging you?

#283

Post by dap »

FridgeSeal wrote: post #246
dap wrote: post #192

Earlier quoted context omitted.

Thanks -- that's very clarifying! But isn't this circular? The lack of bugs is used as evidence of the effectiveness of the testing approach, but the testing approach is validated by...not finding any more bugs in the software?

Yeah but if your software is running in an environment that controls for a lot of non-determinism and can simulate various kinds of failures and degradations at varying rates, and do it all in accelerated time and your software is still working correctly; I think it’d be somewhat reasonable to assert that maybe the testing setup has done a pretty good job.

Agreed, the approach sounds very interesting and I can see how it could be very effective! I'd love to try it on my own stuff. That's why it's so surprising (to me) to claim that the approach found nearly every bug in something as complicated as a production distributed database. My career experience tells me (quite strongly) that can't possibly be true.

Re: Is something bugging you?

#284

Post by criddell »

wwilson wrote: post #95
foobarqux wrote: post #73

Earlier quoted context omitted.

Except it doesn't actually explain in what it does: Is it fuzzing? Do you supply your own test cases? Is it testing hardware non-determinism?

Post author here. Sorry it was vague, but there's only so much detail you can go into in a blog post aimed at general audiences. Our documentation ( https://antithesis.com/docs/ ) has a lot more info. Here's my attempt at a more complete answer: think of the story of the blind men and the elephant. There's a thing, called fuzzing, invented by security researchers. There's a thing, called property-based testing, inven…

> turn them into something really usable for the vast majority of software

Would it work for debugging, say, Notepad on Windows?

Re: Is something bugging you?

#285

Post by amw-zero »

wwilson wrote: post #95
foobarqux wrote: post #73

Earlier quoted context omitted.

Except it doesn't actually explain in what it does: Is it fuzzing? Do you supply your own test cases? Is it testing hardware non-determinism?

Post author here. Sorry it was vague, but there's only so much detail you can go into in a blog post aimed at general audiences. Our documentation ( https://antithesis.com/docs/ ) has a lot more info. Here's my attempt at a more complete answer: think of the story of the blind men and the elephant. There's a thing, called fuzzing, invented by security researchers. There's a thing, called property-based testing, inven…

Is there more info on how Antithesis solves problem number 2 (large state spaces)? I understand the fuzzing / workload generation part well, but there's so many different state space reduction techniques that I don't know what Antithesis is doing under the hood to combat that.

Re: Is something bugging you?

#286

Post by pkghost »

ajb wrote: post #209
wwilson wrote: post #95

Earlier quoted context omitted.

Post author here. Sorry it was vague, but there's only so much detail you can go into in a blog post aimed at general audiences. Our documentation ( https://antithesis.com/docs/ ) has a lot more info. Here's my attempt at a more complete answer: think of the story of the blind men and the elephant. There's a thing, called fuzzing, invented by security researchers. There's a thing, called property-based testing, inven…

This is interesting - it is kind of picking a fight with SaaS/cloud providers though, as that is the one kind of software you won't be able to import into your environment: not because it can't do the job, but because you don't have the code. So this would create an incentive to go back to PaaS. It's definitely true though that a big problem with backend is that you can't easily treat it as a whole system for test pu…

> it is kind of picking a fight with SaaS/cloud providers

or starting a bidding war

Re: Is something bugging you?

#287

Post by hintymad »

I really like antithesis' approach: it's non-intrusive as all the changes are on a VM so one can run deterministic simulation without changing their code. It's also technically challenging, as making a VM suitable for deterministic simulation is not an easy feat.

On a side, I was wondering how this approach compares to Meta's Hermit(https://github.com/facebookexperimental/hermit), which is a deterministic Linux instead of a VM.

Re: Is something bugging you?

#288

Post by Voultapher »

indiv0 wrote: post #10

I've been super interested in this field since finding out about it from the `sled` simulation guide [0] (which outlines how FoundationDB does what they do). Currently bringing a similar kind of testing in to our workplace by writing our services to run on top of `madsim` [1]. This lets us continue writing async/await-style services in tokio but then (in tests) replace them with a deterministic executor that patches…

> you can test hundreds of thousands of subtle failure cases in I/O, event ordering, timeouts, dropped packets, filesystem failures, etc.

As cool as all this is, I can't stop but wonder how often the culture of micro-services and distributed computing is ill advised. So much complexity I've seen in such systems boils down to calling a "function" is: async, depends on the OS, is executed at some point or never, always returns a bunch of strings that need to be parsed to re-enter the static type system, which comes with its own set of failure modes. This makes the seemingly simple task of abstracting logic into a named component, aka a function, extremely complex. You don't need to test for any of the subtle failures you mentioned if you leave the logic inside the same process and just call a function. I know monoliths aren't always a good idea or fit, at the same time I'm highly septical whether the current prevalence of service based software architectures is justified and pays off.

Re: Is something bugging you?

#289

Post by _a_a_a_ »

SomeCallMeTim wrote: post #239
didgetmaster wrote: post #113

Earlier quoted context omitted.

It seems like the industry would get a lot more 10x behavior if it was recognized and rewarded more often than it currently does. Too often, management will focus more on the guy who works 12 hour days to accomplish 8 hours of real work than the guy who gets the same thing accomplished in an 8 hour day. Also, deviations from 'normal' are frowned upon. Taking time to improve the process isn't built into the schedule;…

This reminds me of the "Parable of the Two Programmers." [1] A story about what happens to a brilliant developer given an identical task to a mediocre developer. [1] I preserved a copy of it on my (no-advertising or monetization) blog here: https://realmensch.org/2017/08/25/the-parable-of-the-two-pro...

Without more backup I can only describe that as being fiction. Righteous fiction, where the good guy gets downtrodden and the bad guy wins to fuel the reader's resentment.

Re: Is something bugging you?

#290

Post by chinchilla2020 »

mamidon wrote: post #199

I can see how determinism can be achieved (not easy, but possible), and I can see how describing a few important system invariants can match 100's or 1000's of hand rolled tests, but I'm having a hard time understanding how it's possible to intelligently explore the inputs to generate. e.g. if I wrote a compiler, how would Antithesis generate mostly valid source code for it? Simply fuzzing utf8 inputs wouldn't get ve…

The blog post has some impressive copy but is lacking details on how you implement their product.

I am highly skeptical of any claims that something 'magically just works' without much configuration or setup.

Post reply on HN